New around here? Register your SA Forums Account here!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
mystes
May 31, 2006

Symbolic Butt posted:

the weirdest thing about han unification is how they didn't even bother to put a language variant tag/modifier (like you can specify skintones with a modifier on emojis nowadays?)
As I said earlier, they do have variation selectors for the different forms of the characters now.

Adbot
ADBOT LOVES YOU

Symbolic Butt
Mar 22, 2009

(_!_)
Buglord

pseudorandom name posted:

they introduced the language tags in Unicode 3.1, deprecated them in 5.1, and removed them entirely in 8

mystes posted:

As I said earlier, they do have variation selectors for the different forms of the characters now.

ok so it's still a confusing mess? yeah, seems like han unification was a great technical decision you guys...

pseudorandom
Jun 16, 2010



Yam Slacker
this discussion is getting out of han

redleader
Aug 18, 2005

Engage according to operational parameters

ty for the additional context and details

TheFluff
Dec 13, 2006

FRIENDS, LISTEN TO ME
I AM A SEAGULL
OF WEALTH AND TASTE

Symbolic Butt posted:

ok so it's still a confusing mess? yeah, seems like han unification was a great technical decision you guys...

trying to model natural language on a computer is terrible programming in and of itself

Aramoro
Jun 1, 2012




Powerful Two-Hander posted:

it's a search complexity problem really, if I search for a value in column X and sort by column Y the whole dataset has to be sorted so a search is always gonna be slow. what I can do though is push the generic "default" search into a different procedure that doesn't need to plan for variables so would return a lot quicker (there is still some variation due to per user access permissions though). its a bit messy though because it means having two procedures returning the same data and testing searches to see if they're a "search with no prior" or "search for filter values that are all empty"


caching is also an option as the data returned is json.... but our iis farms are behind a proxy managed by another team who generally refuse to do anything helpful like configure it

edit: tbh I know this is the real answer rather than caching results in session.

Search performance would have to be crippling for this to be a good idea surely? Providing 2 paths to return search results is a maintenance headache. If it is just for sorting then do it in the client unless you're paging the results?

I just can't think of any situation I would want to cache search results on the server.

MononcQc
May 29, 2007

TheFluff posted:

trying to model natural language on a computer is terrible programming in and of itself

just be grateful nobody decided to just throw tensorflow at it as part of the Unicode committee

Powerful Two-Hander
Mar 9, 2004

Mods please change my name to "Tooter Skeleton" TIA.



Aramoro posted:

Search performance would have to be crippling for this to be a good idea surely? Providing 2 paths to return search results is a maintenance headache. If it is just for sorting then do it in the client unless you're paging the results?

I just can't think of any situation I would want to cache search results on the server.

it's not crippling just laggy, like 4-5 seconds on load and the search is the landing page so every time you go back you've gotta wait for it to rerun your previous search to render the grid content

sorting in the client would mean streaming 110k rows from the db to the server, sorting them then throwing 99.9% away to get the first 50 for the page so that's madness.

idk the lazy option is use a different query for the basic "no filter" load which will then be more or less instant, caching the last results client side would be better though as they are just json returned to a grid plug in

Aramoro
Jun 1, 2012




Powerful Two-Hander posted:

it's not crippling just laggy, like 4-5 seconds on load and the search is the landing page so every time you go back you've gotta wait for it to rerun your previous search to render the grid content

sorting in the client would mean streaming 110k rows from the db to the server, sorting them then throwing 99.9% away to get the first 50 for the page so that's madness.

idk the lazy option is use a different query for the basic "no filter" load which will then be more or less instant, caching the last results client side would be better though as they are just json returned to a grid plug in

4-5 seconds is pretty crippling tbh, that's just under a general users boredom threshold.

What would be your strategy for when to burst the cache on the search results? That would be a big bit for me, how out of date can the results be. Your use case here might make caching a good strategy. So the results set is 110K rows and you're just displaying top 50, can you just top the query and reduce that? How interactive is the results set the user can see? Can they sort it etc requiring re-fetching the results? Can they page through the results?

If you do cache the results put some indication on the UI at least as to when the query was first run at least.

Powerful Two-Hander
Mar 9, 2004

Mods please change my name to "Tooter Skeleton" TIA.



Aramoro posted:

4-5 seconds is pretty crippling tbh, that's just under a general users boredom threshold.

What would be your strategy for when to burst the cache on the search results? That would be a big bit for me, how out of date can the results be. Your use case here might make caching a good strategy. So the results set is 110K rows and you're just displaying top 50, can you just top the query and reduce that? How interactive is the results set the user can see? Can they sort it etc requiring re-fetching the results? Can they page through the results?

If you do cache the results put some indication on the UI at least as to when the query was first run at least.

it's got it all: sorting, paging filtering and users can have different permissions on what they can see. the permissions are a pain in the rear end because they mean you can't just cache the render result for the basic no filter search

currently every query hits the database and though the query is pretty well optimised, it's still got to do a load of work. idk I'll have more of a think about what it makes sen to cache and where

good point on the aging of it. I'll add a refresh button titled "blow it out your cache"

distortion park
Apr 25, 2011


idk if this would work in your case, but if it's an internal site you might be able to get away with displaying top 50 in client on their initial search, then streaming the rest to the client in the background so that you don't have to do any caching serverside. 110k rows might be small enough that you can the just do all the sorting etc clientside.

Aramoro
Jun 1, 2012




pointsofdata posted:

idk if this would work in your case, but if it's an internal site you might be able to get away with displaying top 50 in client on their initial search, then streaming the rest to the client in the background so that you don't have to do any caching serverside. 110k rows might be small enough that you can the just do all the sorting etc clientside.

This is kinda what we do. We have user home pages with searches on them, you can have a few of them, and you can defined which columns you see etc. You get your top 25 on each query when you login and you can optionally ask it to show you more or go to the proper search screen if you want to do more.

Perhaps a bit of analytics on the user behaviour might help on working out your solution. Like how often do users sort the data, page through it etc. If it's like us then it's never sort, never page because they've already setup their query to show the data they want to display.

Depending on the width of the dataset then sorting could definitely be done in the client if you can get the data there sensibly.

Wheany
Mar 17, 2006

Spinyahahahahahahahahahahahaha!

Doctor Rope
i have been splitting a java god class with a god method into smaller pieces. i finished that today. now there are 7 classes with each having one largish method that does a lot of processing. that felt good as is, but then i made one additional change:

i had a mehtod with this kind of signature
code:
@Transactional 
public void processButts(List<Butt> butts) {
}
when doing a full run, that processes about 15k items.
the whole program ran in a single transaction (unnecessarily) and this method ran at the end. the program took about 2 hours to finish when doing a full run. (usually we just process data that has changed since the last run, but even that took close to 10 minutes)

after i separated all these methods into their own classes, that also separated the transactions, because the main method that calls all of these methods no longer needed to be @Transactional. that cut down the processing time for this monster method to under an hour.

then because i got a hunch, i made a loop in the main method to feed the method 50-element sublists from the original list, each of them in their own separate transaction.

that cut the processing time of the whole program doing a full run down to 5 minutes. loving what?

the original horrible code was written during the february crunch i have mentioned before. just piling poo poo code on top of old poo poo code because the specs keep changing.
then i have had to put out fires and build workarounds for a couple of months because the code performed like poo poo in production.
then in may there was another two-week emergency and cleaning out the fallout from that.
now i finally had a couple of weeks of relatively chill time to clean up the mess from february and that cuts the worst case processing time from 2 hours to 5 minutes, which just... completely eliminates any performance issues on our end.

Wheany
Mar 17, 2006

Spinyahahahahahahahahahahahaha!

Doctor Rope
the hunch was because i had added progress reporting to the log, and it went
processed 100 items, estimated time left 1m 12s
processed 150 items, estimated time left 3m 40s
processed 175 items, estimated time left 5m 30s
the number of processed items kept getting smaller in each time window and the estimate kept growing until it stabilized to about 6 minutes where it stayed for about 30 minutes before it started decreasing.

animist
Aug 28, 2018

lmao

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


i'm searching almost 15 year old C memory here. do execv() and friends do anything to the process space's heap? or is that platform/impl dependent

i'm chasing a memory corruption bug that only appears when this library emergency-restarts, and the final call is just a plain old forkless execv. i've not seen this before, and i think it's causing an uninitialized pointer elsewhere to look initialized after the image is loaded and starts running, if that makes sense

code:
      execv(g_hMotionWnd, "\\hard disk\\boamotion", gMotionSettings->argv);
(it also seems like a "clever" way to restart a program, scare quotes intended, but idk on that one, might be standard procedure)

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat
execv should replace the entire process, the old one will get terminated. all that memory gets reclaimed by the OS.

your code looks like windows stuff so im assuming the implementation is broken and dumb as all hell.

also how is it "looking" initialized?

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


CRIP EATIN BREAD posted:

execv should replace the entire process, the old one will get terminated. all that memory gets reclaimed by the OS.

your code looks like windows stuff so im assuming the implementation is broken and dumb as all hell.

it's the winCE section, yeah. old product, but it's still got customers and the bug's a P1 :sigh:

that's what I thought happened but thanks for the confirmation

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat
ctps: wanted to use some third party application that only supports a few auth mechanisms one of which was LDAP, so I wrote up a small server based on apache ds to implement my own in-memory LDAP to fake LDAP bind auth requests and hit the database to check the username/password hash.

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat

Ciaphas posted:

it's the winCE section, yeah. old product, but it's still got customers and the bug's a P1 :sigh:

that's what I thought happened but thanks for the confirmation

my guess is that the start-up isnt memzeroing poo poo out, and when allocating some struct it's just junk data.

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


i'm 75% sure all these globals are being used in threaded contexts without enough synch and i'm groaning at having to learn this

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat
just wrap everything with

EnterCriticalSection(&CriticalSection);

and

LeaveCriticalSection(&CriticalSection);

hth

pseudorandom
Jun 16, 2010



Yam Slacker

Ciaphas posted:

i'm searching almost 15 year old C memory here. do execv() and friends do anything to the process space's heap? or is that platform/impl dependent

i'm chasing a memory corruption bug that only appears when this library emergency-restarts, and the final call is just a plain old forkless execv. i've not seen this before, and i think it's causing an uninitialized pointer elsewhere to look initialized after the image is loaded and starts running, if that makes sense

code:
      execv(g_hMotionWnd, "\\hard disk\\boamotion", gMotionSettings->argv);
(it also seems like a "clever" way to restart a program, scare quotes intended, but idk on that one, might be standard procedure)

Is the program crashing on that line? If not, is it happening in similar areas in your program?

Is the memory being corrupted in the memory of the main process, or the memory of the library? My complete shot in the dark guess is the library might be returning (or consuming) a pointer that's being held on to which shouldn't be.

Kazinsal
Dec 13, 2011

are you actually doing a fork at some point in that code or are you just using exec to somehow overwrite your own process?

because while you should really be doing neither in windows, at least the former is sort of sensible

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


pseudorandom posted:

Is the program crashing on that line? If not, is it happening in similar areas in your program?

Is the memory being corrupted in the memory of the main process, or the memory of the library? My complete shot in the dark guess is the library might be returning (or consuming) a pointer that's being held on to which shouldn't be.

Turns out the library was being restarted right AFTER it wrote an inventory swap to disk, but before the robot finished the maneuver, and the bug was in the startup inventory reconciliation (missing memset as CRIP EATIN BREAD predicted)

So I can fix the reconciliation; the problem is I was never actually able to reproduce the bug--some details were missing in the original report, and try as I might I haven't been able to suss them out enough to even get a 10% repro [there's a race condition involved at the exact moment of the restart, I think]

prisoner of waffles
May 8, 2007

Ah! well a-day! what evil looks
Had I from old and young!
Instead of the cross, the fishmech
About my neck was hung.
totally worth it to dig deep and find your best authoritative docs in some MSDN CD set in molding cardboard.

prisoner of waffles
May 8, 2007

Ah! well a-day! what evil looks
Had I from old and young!
Instead of the cross, the fishmech
About my neck was hung.

Ciaphas posted:

Turns out the library was being restarted right AFTER it wrote an inventory swap to disk, but before the robot finished the maneuver, and the bug was in the startup inventory reconciliation (missing memset as CRIP EATIN BREAD predicted)

So I can fix the reconciliation; the problem is I was never actually able to reproduce the bug--some details were missing in the original report, and try as I might I haven't been able to suss them out enough to even get a 10% repro [there's a race condition involved at the exact moment of the restart, I think]

holy poo poo

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


Kazinsal posted:

are you actually doing a fork at some point in that code or are you just using exec to somehow overwrite your own process?

because while you should really be doing neither in windows, at least the former is sort of sensible

no fork; it's blowing itself away and starting over. the equivalent of ctrl-c and retyping the command, I suppose

in this case that execv is happening because a robot axis being controlled by the library (one of the robot axes) reinitialized (power loss), and I guess restarting the library was the solution they went with

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


I've not encountered this professionally, somehow, so I'll just ask


when you've got a bug like this that you think you've put in enough defensive code to prevent relapse, BUT for whatever reason you can't reproduce the original bug in the first place, how is my time best spent going forward

- find the drat bug come hell or high water, doesn't matter how unlikely it is to produce in the Real World
- code defensively, put in traces, keep an eye out, and hope like hell it doesn't happen again while I get some other work done


I guess this is more a 'depends on your manager' question, huh

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat
i'm just gonna assume that thing is using named shared memory as well.

sorry for your loss

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


CRIP EATIN BREAD posted:

i'm just gonna assume that thing is using named shared memory as well.

sorry for your loss

yep. it has an option to use a file for inventory instead but by default it heads for either /dev/slram or /dev/nvram, whichever is available, and that was the case here

the best part is, when it's using a file, it communicates that to the rest of the system over IP, and I would have seen the contents in the submitted bug logs. Here, because it went through shared memory, and that wasn't logged because of local space constraintt, I have to guess

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat
if you put enough defensive poo poo in, just write to a log somewhere.

i haven't touched wince in forever, can you dump a core on failure and then debug it later? if so, enable that.

my boss also trusts me to do whatever necessary to get poo poo done (no gods, no managers here), but if it were me i'd just set up something that dumps the core and keeps the last X around.

CRIP EATIN BREAD
Jun 24, 2002

Hey stop worrying bout my acting bitch, and worry about your WACK ass music. In the mean time... Eat a hot bowl of Dicks! Ice T



Soiled Meat

Ciaphas posted:

yep. it has an option to use a file for inventory instead but by default it heads for either /dev/slram or /dev/nvram, whichever is available, and that was the case here

lol called it. i think you are me in some weird time loop offset a decade in the past. hello, my past self.

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


CRIP EATIN BREAD posted:

lol called it. i think you are me in some weird time loop offset a decade in the past. hello, my past self.

let me know what the winning lotto numbers are for colorado in july, will ya

Bloody
Mar 3, 2013

Ciaphas posted:

I've not encountered this professionally, somehow, so I'll just ask


when you've got a bug like this that you think you've put in enough defensive code to prevent relapse, BUT for whatever reason you can't reproduce the original bug in the first place, how is my time best spent going forward

- find the drat bug come hell or high water, doesn't matter how unlikely it is to produce in the Real World
- code defensively, put in traces, keep an eye out, and hope like hell it doesn't happen again while I get some other work done


I guess this is more a 'depends on your manager' question, huh

option 2. if option 2 isnt viable, then thread title

Bloody
Mar 3, 2013

unless its software where it actually matters, like say you work on mcas

Arcsech
Aug 5, 2008

Ciaphas posted:

I've not encountered this professionally, somehow, so I'll just ask


when you've got a bug like this that you think you've put in enough defensive code to prevent relapse, BUT for whatever reason you can't reproduce the original bug in the first place, how is my time best spent going forward

- find the drat bug come hell or high water, doesn't matter how unlikely it is to produce in the Real World
- code defensively, put in traces, keep an eye out, and hope like hell it doesn't happen again while I get some other work done


I guess this is more a 'depends on your manager' question, huh

depends on this impact of the bug and the value of the work you could otherwise be getting done

assess the potential impact of the bug, talk to your manager about it and ask what they want you to do. that's literally their job: to translate between engineering and business concerns

e: if they tell you to leave it and work on other poo poo, get it in writing, especially if the impact of the bug is bad

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


Arcsech posted:

depends on this impact of the bug and the value of the work you could otherwise be getting done

assess the potential impact of the bug, talk to your manager about it and ask what they want you to do. that's literally their job: to translate between engineering and business concerns

e: if they tell you to leave it and work on other poo poo, get it in writing, especially if the impact of the bug is bad

inventory corruption (in this case, the inventory showing a single tape both in the robot picker head AND in a magazine slot, despite the robot being empty & available for moves) bugs are always P1

BUT

afaict the picker head inventory contents don't actually matter to the customer that the inventory is being reported to, because a tape being in the robot implies a move hasn't finished, and inventories taken while a move is incomplete are considered suspect. (In fact, other versions of the product don't even report the robot inventory for that reason--just the real storage slots--so that might be enough cause to move on all itself, now I think about it)

Ciaphas fucked around with this message at 22:26 on Jun 18, 2019

Mahatma Goonsay
Jun 6, 2007
Yum

Ciaphas posted:

I've not encountered this professionally, somehow, so I'll just ask


when you've got a bug like this that you think you've put in enough defensive code to prevent relapse, BUT for whatever reason you can't reproduce the original bug in the first place, how is my time best spent going forward

- find the drat bug come hell or high water, doesn't matter how unlikely it is to produce in the Real World
- code defensively, put in traces, keep an eye out, and hope like hell it doesn't happen again while I get some other work done


I guess this is more a 'depends on your manager' question, huh

as someone who used to have to write software to interface to incredibly unreliable lab equipment option b is sometimes the only solution

Adbot
ADBOT LOVES YOU

Ciaphas
Nov 20, 2005

> BEWARE, COWARD :ovr:


oops did that thing where i ramble mostly to myself in a post

well, useful notes for tomorrow at least :D

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply