Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Smol
Jun 1, 2011

Stat rosa pristina nomine, nomina nuda tenemus.

no_funeral posted:

i'm mostly worried about the production environment, which is a unicorn server. since unicorn is process based, i don't think it will be an issue?

It will be an issue. Just get rid of the time zone change in the first place (sounds like a huge-rear end code smell), or make sure that you clean that poo poo up.

Smol fucked around with this message at 11:18 on Nov 7, 2015

Adbot
ADBOT LOVES YOU

necrotic
Aug 2, 2005
I owe my brother big time for this!

Smol posted:

It will be an issue. Just get rid of the time zone change in the first place (sounds like a huge-rear end code smell), or make sure that you clean that poo poo up.

Yup. Once you change the Time.zone for the process it will remain changed until you reset it. We wrap our user-centric actions in an around_filter that sets/unsets the Time.zone to where the user specified.

Mo_Steel
Mar 7, 2008

Let's Clock Into The Sunset Together

Fun Shoe
Learning Ruby for the first time working through the Codecademy stuff, and I got to the sorting section. It almost broke my brain for a bit trying to figure out how the hell things were actually working, and it occurs to me maybe my pain would be helpful. When sorting an array, the .sort method uses the combined comparison comparison or "spaceship" operator (which looks like <=> ) to compare two elements of the array at a time returning -1, 0 , or 1:
code:
n1 <=> n2
If n1 < n2, the comparison returns -1
If n1 == n2, the comparison returns 0
If n1 > n2, the comparison returns 1

Why does this matter for sorting? Sort compares the two elements and does the following:
    If the comparison returns 1 it swaps their position in the array
    If the comparison returns 0 or -1 it doesn't swap their positions

The swapping positions part was what was tripping me up in understanding sorting; because I was incorrectly reading it like this:
    If the comparison returns -1, the number on the left should be first
    If the comparison returns 1, the number on the left should be second

My confusion was initially compounded because this incorrect reading technically works, but only for ascending order. When I tried it in descending order I struggled trying to follow the steps because I was making the wrong step in how sort uses the returned comparison.

For example:

code:
my_array = [6, 17, 9]

my_array.sort! { |n1, n2|  
    n1 <=> n2 }

# 6 <=> 17 returns -1, so the numbers are not swapped and the array is now [6, 17, 9]; now make n1 = 17 and n2 = 9
# 17 <=> 9 returns 1, so the numbers are swapped and the array is now [6, 9, 17]
If you want them in reverse order, then you reverse the order of the values being compared:

code:
my_array = [6, 17, 9]

my_array.sort! { |n1, n2|  
    n2 <=> n1 }

# 17 <=> 6 returns 1, so the numbers are swapped and the array is now [17, 6, 9]; now make n1 = 6 and n2 = 9
# 9 <=> 6 returns 1, so the numbers are swapped and the array is now [17, 9, 6]
-------------

I'm totally new at Ruby so if my understanding is still woefully incorrect please fell free to correct me; I know the sort method steps through things in a bit more complicated manner than that as it clearly iterates through the array more than one time to make sure sorting is completed.

MrPablo
Mar 21, 2003

Mo_Steel posted:

I'm totally new at Ruby so if my understanding is still woefully incorrect please fell free to correct me; I know the sort method steps through things in a bit more complicated manner than that as it clearly iterates through the array more than one time to make sure sorting is completed.

Ruby uses Quicksort under the hood to sort Arrays (and Enumerables):


Quicksort is O(n * log(n)) on average and O(n^2) worst case. In english, that means Quicksort can sort a list with 10 elements in about 23 steps on average and 100 steps at worst. That's pretty good for a sorting algorithm, which is why Quicksort is extremely common.

Have a look at the Algorithm section of the Quicksort Wikipedia page for a basic explanation of the algorithm.

MrPablo fucked around with this message at 02:32 on Nov 9, 2015

KoRMaK
Jul 31, 2012



EVGA Longoria posted:

At my current company, we do SE 1-3, Sr SE and then Architect. Lead is another path that replaces Senior/Architect. It makes sense to me, so that's how I'd think. At 10+ years, I would expect so one to be applying for one or the other explicitly.

Whats the diff?

To me, Lead has some management stuff going on - people stuff.

kayakyakr
Feb 16, 2004

Kayak is true

KoRMaK posted:

Whats the diff?

To me, Lead has some management stuff going on - people stuff.

Lead does have management stuff. Not everyone wants to do management.

Mo_Steel
Mar 7, 2008

Let's Clock Into The Sunset Together

Fun Shoe

MrPablo posted:

Ruby uses Quicksort under the hood to sort Arrays (and Enumerables):


Quicksort is O(n * log(n)) on average and O(n^2) worst case. In english, that means Quicksort can sort a list with 10 elements in about 23 steps on average and 100 steps at worst. That's pretty good for a sorting algorithm, which is why Quicksort is extremely common.

Have a look at the Algorithm section of the Quicksort Wikipedia page for a basic explanation of the algorithm.

Cool stuff, a pivot value never occurred to me but I can see how that would cut down on time to sort compared with just repeatedly checking all values with their neighbors until you never get a swap response.

Pollyanna
Mar 5, 2005

Milk's on them.


Let's say something went wrong in production and you have to do some database surgery through ActiveRecord. What's the safest, most accepted way of hotfixing live, production data?

The original method I was taught, "create a Ruby script and load it from Rails console while pointing to our production database", just doesn't sit right with me. We have no canonical way of hotfixing our database issues, though, so in lieu of anything else, that's the default approach. Is there something better?

KoRMaK
Jul 31, 2012



Got a backup?

I have been known at times to log into the production console and do stuff.


Or, write a migration.

Peristalsis
Apr 5, 2004
Move along.

Pollyanna posted:

Let's say something went wrong in production and you have to do some database surgery through ActiveRecord. What's the safest, most accepted way of hotfixing live, production data?

The original method I was taught, "create a Ruby script and load it from Rails console while pointing to our production database", just doesn't sit right with me. We have no canonical way of hotfixing our database issues, though, so in lieu of anything else, that's the default approach. Is there something better?

For a small, one-off fix that just needs to get done, I'd be awfully tempted just to use rails console, with an appropriate amount of caution.

kayakyakr
Feb 16, 2004

Kayak is true
Yeah, I'll usually do that through rails console, just make sure you don't gently caress it up.

If it's a bigger batch of changes, you can write a migration or do it through db:seed.

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home
Yeah, depends on the scope of the adjustments. Super simple one-liners or just touching a handful of records (if this is the case make sure you're only touching these records!) dropping into production console is fine. Just 1) backup first, 2) be careful.

If it's something more involved I will write a one-off rake task and test it against a copy of production first

Pollyanna
Mar 5, 2005

Milk's on them.


Sounds good to me. I made a spec that mimicks our current issue and from what I can tell, the script/class/module/thing-I-made handles it correctly. I guess logging into console works well. I tested it on my local copy of our DB, and it works there too.

We don't do production DB backups at my place, though, as far as I know. At least, not regularly. Something about it being too much data, or whatever.

MasterSlowPoke
Oct 9, 2005

Our courage will pull us through
It's too much data, so if it's lost then it's easier to replace than a backup hard drive?

aunt jenkins
Jan 12, 2001

Pollyanna posted:

Let's say something went wrong in production and you have to do some database surgery through ActiveRecord. What's the safest, most accepted way of hotfixing live, production data?

Home projects, Rails console.
At work, has to be a migration, with the usual code review and testing etc that accompanies any other production code change.

KoRMaK
Jul 31, 2012



Pollyanna posted:

We don't do production DB backups at my place, though, as far as I know. At least, not regularly. Something about it being too much data, or whatever.
Lol.

Thalagyrt
Aug 10, 2006

Pollyanna posted:

Sounds good to me. I made a spec that mimicks our current issue and from what I can tell, the script/class/module/thing-I-made handles it correctly. I guess logging into console works well. I tested it on my local copy of our DB, and it works there too.

We don't do production DB backups at my place, though, as far as I know. At least, not regularly. Something about it being too much data, or whatever.

We have a 1.5TB database at my day job that's fully backed up to S3 daily... What's your excuse for not caring about your business's data at all?

Thalagyrt fucked around with this message at 20:21 on Nov 11, 2015

KoRMaK
Jul 31, 2012



Maybe they don't have a station wagon big enough.

Thalagyrt
Aug 10, 2006

KoRMaK posted:

Maybe they don't have a station wagon big enough.

lol. never underestimate the bandwidth of a station wagon full of hard drives

Sewer Adventure
Aug 25, 2004

Peristalsis posted:

For a small, one-off fix that just needs to get done, I'd be awfully tempted just to use rails console, with an appropriate amount of caution.

One time in the production console, I wanted to filter an array of all users to find those that belonged to a certain team.

code:
users = User.all
team = Team.find(123)
members = users.select {|u| u.team = team}
Because it was a has_one relation, assignment doesn't need a save call :pseudo:

Pardot
Jul 25, 2001




Ahhh please everyone if you don't use a Postgres service that sets up wal shipping for you so you have point in time recovery, please set that up right now.

Sewer Adventure
Aug 25, 2004

Pardot posted:

Ahhh please everyone if you don't use a Postgres service that sets up wal shipping for you so you have point in time recovery, please set that up right now.

Yeah, my sysadmin saved me using something like that. This was 5 years ago though.

necrotic
Aug 2, 2005
I owe my brother big time for this!

Pardot posted:

Ahhh please everyone if you don't use a Postgres service that sets up wal shipping for you so you have point in time recovery, please set that up right now.

What about MySQL? :saddowns:

Also we do migrations that go through the normal review process for any data changes like that.

MasterSlowPoke
Oct 9, 2005

Our courage will pull us through

Sewer Adventure posted:

Because it was a has_one relation, assignment doesn't need a save call :pseudo:

When I ever have to use the production console like that I make sure to use sandbox mode.

Mo_Steel
Mar 7, 2008

Let's Clock Into The Sunset Together

Fun Shoe
So, quick question; I came from a background of having done some code in C#, and I'm looking at the "yield" functionality in Ruby, like this:

code:
def double_number(num)
    yield(num)
end

double_number(10) { |n| n*2 }
It looks like it basically pulls the block of code in where "yield" is and runs it as though it were structured like this:

code:
def double_number(num)
  num*2
end

double_number(10)
But I'm struggling with the concept of how this is useful. Can anyone recommend or give a simple example where it makes sense to pass a block into a method instead of just having the code already present in the method? Is it more about allowing methods to be flexible with regards to inputs and functionality without having a thousand lines of switch-case and if-else statements and instead just pass in unique blocks if they are needed?

Mo_Steel fucked around with this message at 05:08 on Nov 25, 2015

necrotic
Aug 2, 2005
I owe my brother big time for this!
code:
File.open('foo.txt'. 'w') do |io|
  io.puts('bar')
end # file is closed

# compute a hash value from a key if its not set
hash = Hash.new { |key,hash| hash[key] = do_something_with(key) }
hash[:foo] # calls do_something_with(:foo)
There are more use cases but those are some two good ones.

necrotic fucked around with this message at 05:31 on Nov 25, 2015

Smol
Jun 1, 2011

Stat rosa pristina nomine, nomina nuda tenemus.
Ruby basically allows every method to take a block (think of it as an anonymous function) as a parameter. "yield" simply calls that block with whatever parameters you give it.

Mo_Steel
Mar 7, 2008

Let's Clock Into The Sunset Together

Fun Shoe
Alright, thanks for the additional information.

Thalagyrt
Aug 10, 2006

Mo_Steel posted:

Alright, thanks for the additional information.

I think one of the best examples of the usefulness of yield is Enumerable#each. It's such a simple thing, and so perfectly elegant once you realize what's going on.

code:
[1, 2, 3].each do |num|
  puts num
end
You're passing a block to Enumerable#each here, taking an argument, and printing that argument. It's left up to Enumerable#each to call that block with each element in the enumerable. Rather than iterating over a list by looking at its size and asking it for each element, yield allows you to tell a list to call your code with each element. The key difference here is you're telling the list "Do this" instead of asking the list "Gimme an element. Gimme next element" etc. This offers better encapsulation, as the only thing that ever needs to know how to iterate over an Array is an Array.

From there, I'm sure you can start to see other uses for yield. Another great example is structuring markup languages.

code:
div do
  ul do
    li { "Item a" }
    li { "Item b" }
  end
end
This kind of syntax is again made possible by yielding - you capture the content outputted by the yield and render it as content for the element. It's super powerful and gives the language immense flexibility.

KoRMaK
Jul 31, 2012



Mo_Steel posted:


But I'm struggling with the concept of how this is useful. Can anyone recommend or give a simple example where it makes sense to pass a block into a method instead of just having the code already present in the method?
https://www.youtube.com/watch?v=OMPfEXIlTVE

I'm still reeling from this talk

Peristalsis
Apr 5, 2004
Move along.
I'm not sure if this is a Rails issue, a VM issue, or a MySQL issue, but I'll start here and hope for the best.

Our app has a Datafile model, which just stores OS file metadata, and a DatafilePermission model, which links datafiles to users who have specific accesses to the files.

In our code is this command:
code:
(Datafile.includes(datafile_permissions: [:user]) - taken_files).sort_by {|e| e.name.downcase }
where taken_files is just an array of Datafile objects. Whenever this command is executed, my dev server freezes up. In fact, when I go into rails console, and just execute
code:
Datafile.includes(:datafile_permissions)
the console freezes, and I have to use kill -9 to remove the process. This doesn't happen for anybody else, and it doesn't happen (so far) in our test or production environments. I'm using Vagrant with a headless Ubuntu VirtualBox VM image on my Windows 7 computer. Other users are using different setups - I'm the only one on Vagrant, and I think I'm the only one using Windows as my base OS. The platform is Rails 3.2.15 and ruby 2.1.1p76 - that should be pretty consistent through all users. It is, at least, the same as test environment.

I bumped the memory allocated in my Vagrantfile from 2 GB up to 12 GB (my PC only has 16, so I can't really go much higher), and it didn't solve the problem. I also destroyed and rebuilt my Vagrant box (which was a pain, as the Vagrantfile is a bit out of date in terms of setting everything up), on the assumption that my VM image might be corrupted, but the problem persists.

My datafiles table has 13,537 rows, and the datafiles_permissions table has just over 18,257 rows. If I limit the output of the query, using
code:
Datafile.includes(:datafile_permissions).limit(5000)
it works up to a limit of almost 5500 rows.

The explain feature doesn't tell me much:
code:
puts Datafile.includes(:datafile_permissions).limit(5000).explain

EXPLAIN for: SELECT  `datafiles`.* FROM `datafiles`  LIMIT 5000
+----+-------------+-----------+------+---------------+------+---------+------+-------+-------+
| id | select_type | table     | type | possible_keys | key  | key_len | ref  | rows  | Extra |
+----+-------------+-----------+------+---------------+------+---------+------+-------+-------+
|  1 | SIMPLE      | datafiles | ALL  | NULL          | NULL | NULL    | NULL | 13028 |       |
+----+-------------+-----------+------+---------------+------+---------+------+-------+-------+
1 row in set (0.00 sec)

...

EXPLAIN for: SELECT `datafile_permissions`.* FROM `datafile_permissions`  WHERE `datafile_permissions`.`datafile_id` IN (1, 2, 3, 4, 5, 6, 
7, 8, 9, 10, 11,...
+----+-------------+----------------------+------+-------------------------------------------+------+---------+------+-------+-------------+
| id | select_type | table                | type | possible_keys                             | key  | key_len | ref  | rows  | Extra       |
+----+-------------+----------------------+------+-------------------------------------------+------+---------+------+-------+-------------+
|  1 | SIMPLE      | datafile_permissions | ALL  | index_datafile_permissions_on_datafile_id | NULL | NULL    | NULL | 18861 | Using where |
+----+-------------+----------------------+------+-------------------------------------------+------+---------+------+-------+-------------+
1 row in set (0.01 sec)
And to_sql doesn't show me much, because the real problem is in the nested query:
code:
[3] pry(main)> x = Datafile.includes(:datafile_permissions).to_sql
=> "SELECT `datafiles`.* FROM `datafiles` "
MySQL's max_allowed_packet variable is set to 16777216, so I don't think it should be choking just because of the long list of IDs in the SQL IN clause. And even if it were a problem, I think I should get an error, not just a freeze. Further, I believe I cobbled together the problematic query (with all 18,000+ ids in parentheses), and it ran fine at the MySQL command line.

So - does anyone have any suggestions? I guess I could try scrapping the includes method, and constructing the list with lower-level commands, but this was changed to use includes to resolve an n+1 queries problem that was bogging things down and making the screens with datafiles pretty much unusable. I'm not finding much of use on Google, and the only other thing I can think of to try is to start updating components (MySQL, Virtual Box, etc.), and hope some update magically fixes this error. I'd like to avoid that, because I still won't know if we have a bit of code that might blow up again at some point, and also because I don't want to end up breaking some gem version dependency just to track down a stupid bug that only affects me to begin with.

Edit: Is it possible that something in the environment settings could cause this? Since this is dev, could there be some setting that causes rails to try to do too much - like extra logging, or not caching something that clogs up the memory?

Edit 2: I replaced development environment file with the production environment file, and nothing changed, so that can't be the issue.

Peristalsis fucked around with this message at 23:07 on Nov 27, 2015

Cock Democracy
Jan 1, 2003

Now that is the finest piece of chilean sea bass I have ever smelled

Peristalsis posted:

Datafile model stuff
What's your intention with

quote:

Datafile.includes(datafile_permissions: [:user])
? I would expect that you're trying to filter Datafiles for a specific user, so there would be a where method in your query, something like:

quote:

Datafile.includes(:datafile_permissions).where('datafile_permissions.user_id = ?', current_user.id).references(:datafile_permissions)
but perhaps it sounds like you want all datafiles and their permissions? If that's the case, I agree that it's not such a crazy amount of data that it should take very long. I'd be interested in seeing the model code for Datafile and DatafilePermission. Maybe they are set up wrong somehow.

necrotic
Aug 2, 2005
I owe my brother big time for this!

Cock Democracy posted:

I'd be interested in seeing the model code for Datafile and DatafilePermission. Maybe they are set up wrong somehow.

It could be the nested loading (Datafile -> Permissions -> Users) causing issues. What happens if you do the same thing in a script, not a Rails console? It could be an issue "rendering" with the deep nesting, I know I've had issues with it before.

Peristalsis
Apr 5, 2004
Move along.

Cock Democracy posted:

What's your intention with

? I would expect that you're trying to filter Datafiles for a specific user, so there would be a where method in your query, something like:

I think the ultimate goal is to show a list of files of a certain type that are not already owned by another user or something (it's not my code). As I said, though, the part that's freezing is just retrieving the total list of datafiles and their permissions.

Cock Democracy posted:

but perhaps it sounds like you want all datafiles and their permissions? If that's the case, I agree that it's not such a crazy amount of data that it should take very long. I'd be interested in seeing the model code for Datafile and DatafilePermission. Maybe they are set up wrong somehow.

The datafile code is pretty messy, just because it's one of our main models. I can post the header info if you like (i.e. all the belongs_to and has_many lines, as well as the validations, I guess), but there will be a lot of irrelevant crap. Also, just to be clear, the issue isn't that it's slow, it's that it stops completely. I think I let it grind overnight once, and it was still stalled the next morning. The server and console also don't respond to ctrl-c once they try this command, and the Linux process (server or console) doesn't respond to the regular kill command, so it's pretty well and truly frozen, not just taking a long time.

necrotic posted:

It could be the nested loading (Datafile -> Permissions -> Users) causing issues. What happens if you do the same thing in a script, not a Rails console? It could be an issue "rendering" with the deep nesting, I know I've had issues with it before.

I'll try a script version of it on Monday, when I get back to work. However, even just one level of nesting is causing the issue (Datafile -> Permissions), but only on my machine.

The last thing I tried before leaving Friday was updating the mysql2 gem, in case I just have a bad version of that. That update, of course, broke everything, so I have to sort that out before I can do anything else. I really don't think it can be an actual rails problem, since it works fine for everyone else's environment (same versions of rails). It also shouldn't be an actual MySQL issue, since the raw query runs okay at a MySQL prompt, so I thought maybe the connector between them was the problem. I also have a different version of MySQL than the test and prod environments have, though mine is more recent.

Peristalsis fucked around with this message at 23:05 on Nov 28, 2015

Peristalsis
Apr 5, 2004
Move along.

Peristalsis posted:

I'll try a script version of it on Monday, when I get back to work. However, even just one level of nesting is causing the issue (Datafile -> Permissions), but only on my machine.

I have a script called includes_script.rb containing the single line
code:
x = Datafile.includes(:datafile_permissions)
When I execute it at a rails console prompt, I just get this:
code:
[6] pry(main)> load 'includes_script.rb'
=> true
Executing this exact same assignment at the command line still freezes the console. However, two things stand out:
1) When I execute the code by loading the script, no SQL stuff shows on the screen. It just thinks for a moment, then returns true. When I do something that actually freezes the console, it tries to show the SQL command as it dies:
code:
[7] pry(main)> x = Datafile.includes(:datafile_permissions)
  Datafile Load (31.0ms)  SELECT `datafiles`.* FROM `datafiles`
  DatafilePermission Load (56.4ms)  SELECT `datafile_permissions`.* FROM `datafile_permissions` WHERE `datafile_permissions`.`datafile_id` IN (1, 2, 3, 4, 5,
  
  ... 
  
  until it just stops, in the middle of an id:
  4974, 4975, 4976, 4977, 4979, 4980, 4981, 49
2) If I add an array typecast to the line in the script - so it now reads
code:
x = Datafile.includes(:datafile_permissions).to_a
- it DOES freeze, (and shows the SQL command that crashes it). There's no explicit typecast in the original, problematic code - it returns an ActiveRecord::Relation object, though it also shows the SQL commands. The array conversion may just be a red herring, but I thought it was worth mentioning.

Peristalsis fucked around with this message at 17:26 on Nov 30, 2015

Smol
Jun 1, 2011

Stat rosa pristina nomine, nomina nuda tenemus.
Try redefining #inspect on the offending classes to something short.

Peristalsis
Apr 5, 2004
Move along.

Smol posted:

Try redefining #inspect on the offending classes to something short.

I guess I don't understand this suggestion. What class should have it overridden, and what will the results of doing so tell me?

Chilled Milk
Jun 22, 2003

No one here is alone,
satellites in every home

Peristalsis posted:

I guess I don't understand this suggestion. What class should have it overridden, and what will the results of doing so tell me?

On the Datafile class, so that the console output is less verbose

Smol
Jun 1, 2011

Stat rosa pristina nomine, nomina nuda tenemus.
Just add "def inspect; "foo"; end" in DataFile and whatever other AR models you were having troubles with. It sounds like the REPL somehow hangs up when it tries to print the results of inspecting the object graph.

Big #inspect messages can be pretty broken especially when something throws a NameError.

Adbot
ADBOT LOVES YOU

necrotic
Aug 2, 2005
I owe my brother big time for this!

Smol posted:


Big #inspect messages can be pretty broken especially when something throws a NameError.

Exactly what I meant with "It could be an issue "rendering" with the deep nesting". The fact that it runs fine in the script is indicative of this as well.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply