cheetah7071 posted:

cheetah7071 posted:

e: this approach also has the advantage that its trivial to restart a run that failed because windows decided to update or something

ultrafilter posted:

Have you looked into spatial databases? I don't know much about those except that they exist, but it's possible that everything you want to do is implemented as standard queries.

ExcessBLarg! posted:

I was going to say, instead of being too concerned about "worrying" memory conditions, maybe implement a checkpointing system so you can restart if you do run out of memory, or for any other reason like the user straight up quitting.

nielsm posted:

I think the worry is that the dataset is already large, and it needs to fit and be processed on a single regular high-end laptop, maybe with an external USB drive. That's why the data is compressed.

ExcessBLarg! posted:

I was going to say, instead of being too concerned about "worrying" memory conditions, maybe implement a checkpointing system so you can restart if you do run out of memory, or for any other reason like the user straight up quitting.

nielsm posted:

Reminds me of the people in the Windows 10 thread crying about it forcing a reboot to install updates, while their scientific processing software is running and the multi-day job is not checkpointed or restartable.

quote:

The desired output is a new grid laid over the same area, with a much finer resolution, where the value in each gridcell is stuff like "the mean z value of all points whose x/y fall within this cell". Basic statistics like that. The resolution of these grids is such that they easily fit in memory on an average machine, but might still be up into the tens of millions of values in a large dataset. In the normal case, where the input files are non-overlapping, the majority of gridcells (~90% of them or so) will only need data from one file. Along the edge between files, they'll need two, and the rare gridcells at the intersection will need four.

Computer viking posted:

Also, and this is not useful in any way -

Am I just missing something, or do you mean a coarser resolution?

cheetah7071 posted:

No I meant finer than the resolution of the input files. The input files cover areas a few square km; I want metrics calculated every few dozen meters.

Wipfmetz posted:

I recently stumbled over a kinda-similar problem, where I've defined some quick & dirty non-const-globals.

https://www.learncpp.com/cpp-tutorial/why-non-const-global-variables-are-evil/ explains it all.

Edit:
I'm not positioning this for a long-term-solution or a solution for constants.

baby puzzle posted:

I want to do something with dates and times but I don't know where to start.

I want to create yearly "events" for which I can specify a month/day/time that they start. Then I need to be able to tell where those events are in relation to the current time. I might even want to do more advanced things like "the first day of spring."

Which library should I be using? Can std::chrono help me do this? I can't really understand anything in there. I'm using C++17.

I may just wind up manually entering timestamps for the events for the next 10 years or something. But if there's a library out there that can make this easy then I'd like to do things properly.

baby puzzle posted:

I want to do something with dates and times but I don't know where to start.

I want to create yearly "events" for which I can specify a month/day/time that they start. Then I need to be able to tell where those events are in relation to the current time. I might even want to do more advanced things like "the first day of spring."

Which library should I be using? Can std::chrono help me do this? I can't really understand anything in there. I'm using C++17.

I may just wind up manually entering timestamps for the events for the next 10 years or something. But if there's a library out there that can make this easy then I'd like to do things properly.

Foxfire_ posted:

Predicting the equinoxes is complicated and involves big tables of observed solar data anyway, so you're probably best off just tabling that

baby puzzle posted:

Thanks for the answers.

You are right about that. I'll just google a few years worth and then wonder why the events stopped happening all of the sudden when the table runs out.

baby puzzle posted:

Thanks for the answers.

You are right about that. I'll just google a few years worth and then wonder why the events stopped happening all of the sudden when the table runs out.

BattleMaster posted:

I have a question for testing a program written in C in Linux that uses system calls.

I have this fully working loop that forks off a bunch of worker processes from the parent:

code:

	for (id = 0; id < args.numWorkers; id++)
	{
		pid = fork();
	
		if (pid == (pid_t)0) // Worker
		{
			break;
		}
		else if (pid < (pid_t)0) // Error
		{
			perror("fork");
			printf("Can't create worker process #%i\n", id);
			
			// Kill processes that have already been spawned
			int numProcesses = id;
			
			for (id = 0; id < numProcesses; id++)
			{
				if (!kill(processTable[id].pid, SIGKILL))
				{
					perror("fork");
				}
			}
			
			exit(EXIT_FAILURE);
		}
		else // Parent
		{
			printf("Spawned worker process %i (PID %i)\n", id, pid);
			
			// Store mapping between our internal id and the process id
			processTable[id].pid = pid;
		}
	}

However, I don't actually know if the part that responds to fork failing actually works because it shouldn't fail under normal conditions - something really messed up needs to be happening for the system to hit maximum processes or not have tens of kilobytes of RAM to spare.

Is there any way I can inject an arbitrary failure into a system call, like saying that after x forks I want the next one to fail?

leper khan posted:

Easiest thing is to have your test code set numWorkers to intmax.