C/C++ Programming Questions Not Worth Their Own Thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > C/C++ Programming Questions Not Worth Their Own Thread

«‹›640 »

seiken: Feb 7, 2005; hah ha ha

nelson posted:

I’ve heard good things about bazel from one guy at our company but I’ve never used it myself. Have any of you used it and if so what is your opinion?

I've used it for a long time now.

It has a fairly steep learning curve, it's very opinionated, and yeah it's no small feat to migrate an existing nontrivial codebase to it.

Dependencies are also not always easy (though it's getting better). rules_foreign_cc allows you to have bazel invoke cmake to build dependencies, but I haven't used it and can't say how well it works. There's also the new bazel central registry, which does let you grab some things with one line, but there's not a ton of stuff there yet. I usually end up writing small custom build files for most dependencies.

If you can deal with all that, you get a hugely powerful, sane, multi-language build system that just works and does the right thing every time. I sometimes run bazel clean just to check everything works from scratch and because I get a strange sense of satisfaction from seeing nice build system outputs churning away, but I can't remember the last time it was actually necessary to fix something. Personally, I wouldn't go back.

seiken fucked around with this message at 12:18 on May 11, 2024

# ? May 11, 2024 10:57

Adbot: ADBOT LOVES YOU

# ? May 25, 2024 00:03

roomforthetuna: Mar 22, 2005; I don't need to know anything about virii! My CUSTOM PROGRAM keeps me protected! It's not like they'll try to come in through the Internet or something!

I prefer old-school Makefiles for having full control over everything, but if I had to use a modern thing, I like bazel a lot more than CMake.

The learning curve to get something just generally working isn't too bad, but once you start getting into needing to be able to import a dependency that isn't something you own, yeah, it can get pretty gnarly. That's probably true of any system.

# ? May 11, 2024 14:53

Subjunctive: Sep 12, 2006; ✨sparkle and shine✨

and bazel�s ability to do a shared artifact cache can be really nice once team and program get bigger. CI tends to keep that cache very warm, so you pretty much only have to compile locally modified stuff ever even after a big pull

# ? May 11, 2024 16:46

pokeyman: Nov 26, 2006; That elephant ate my entire platoon.

I remember more about the couple hours I messed with bazel than the 20 hours I've messed with cmake. Something about cmake is just perfectly designed to slip right off my brain. Makes zero sense to me.

Anyway op cmake is the safe choice. If you try something weird: redo.

# ? May 11, 2024 17:29

Nalin: Sep 29, 2007; Hair Elf

its not a proper c++ build system if you're not doing something weird like using premake5. the joy in life is reverse engineering the build system so you can bolt it on to your own mess

# ? May 12, 2024 01:24

csammis: Aug 26, 2003; Mental Institution

We use waf. It's fine. We have a build tools team to babysit it and I don't have to spend more than a normal amount of mental bandwidth trying to unfuck the tools we depend on for our livelihoods.

# ? May 12, 2024 03:16

giogadi: Oct 27, 2009

In my game engine, I have lots of code for structs that serializes, deserializes, and renders ImGui like the following:

code:

struct FooEntity {
	int hp;
	float maxSpeed;

	void Save(XMLNode* xml) const {
		xml->WriteInt("hp", hp);
		xml->WriteFloat("maxSpeed", maxSpeed);
	}
	void Load(XMLNode const* xml) {
		hp = xml->ReadInt("hp");
		maxSpeed = xml->ReadFloat("maxSpeed");
	}
	void ImGui() {
		ImGui::InputInt("hp", &hp);
		ImGui::InputFloat("maxSpeed", &maxSpeed);
	}
};

However, I'm adding some functionality to diff two entities of the same type, which requires looping over each of these serializable "properties". So I need a way to loop over these members of a struct. I don't have reflection in C++, so I wrote this code that represents Properties as collections of Load/Save/ImGui functions; now I can implement Save/Load/ImGui just one time and it will work for any struct with a corresponding list of Properties:

code:

typedef void (*PropertyLoadFn)(void* v, XMLNode const* xml);
typedef void (*PropertySaveFn)(void const* v, XMLNode* xml);
typedef bool (*PropertyImGuiFn)(char const* name, void* v);
struct PropertySpec {
	PropertyLoadFn loadFn;
	PropertySaveFn saveFn;
	PropertyImGuiFn imguiFn; 
};

void PropertyLoadInt(void* v, XMLNode const* xml) { /* snip */ }
void PropertySaveInt(void const* v, XMLNode* xml) { /* snip */ }
void PropertyImGuiInt(char const* name, void* v) { /* snip */ }

PropertySpec const gPropertySpecInt {
	.loadFn = &PropertyLoadInt,
	.saveFn = &PropertySaveInt,
	.imguiFn = &PropertyImGuiInt
};

/* repeat above for gPropertySpecFloat */

struct Property {
    char const* name;
    PropertySpec* spec; 
    size_t offset;
};

void SaveObject(XMLNode* xml, Property const* properties, int numProperties, void* object) {
	// Go through each property, use Property::offset to access the corresponding member of object, and call its PropertySpec::saveFn
}
void LoadObject(/*snip*/)
void ImGuiObject(/*snip*/)

...and finally, this is all it would take to add Save/Load/ImGui to FooEntity:

code:

struct FooEntity {
	int hp;
	float maxSpeed;
};

Property gFooEntityProperties[] = {
	{
		.name = "hp",
		.spec = &gPropertySpecInt,
		.offset = offsetof(FooEntity, hp)
	},
	{
		.name = "maxSpeed",
		.spec = "&gPropertySpecFloat,
		.offset = offsetof(FooEntity, maxSpeed);
	}
}

I was very proud of this! And this actually works very well for simple C structs. However, when I tried adding it to my engine where some of my classes are derived virtual classes, clang gave me warnings that offsetof should not be used for "non standard layouts"!!!

Is there a way to do what I'm trying above that works safely for classes with vtables? For example, if I replace "offsetof()" with C++ member pointers?

# ? May 14, 2024 13:09

korora: Sep 3, 2011

I think you�re going to hit a dead end with that array of structs because you are going to want heterogeneous types (e.g. member pointers are not uniform types). A solution I have used to reduce boilerplate is to add a function like this to your serialized types:

code:

template <typename S>
void serialize(S serializer) {
    serializer(�hp�, hp, �maxSpeed�, maxSpeed);
}

Then you can write functors to pass into that function that implement your XML save/load and ImGui hooks.

You also shouldn�t need the function pointers for float/int - define those as overloads and let the compiler do the work for you.

# ? May 14, 2024 15:59

giogadi: Oct 27, 2009

(Ugh, I accidentally hit a keyboard shortcut that just posted my message before it was done. Deleting this post)

(LMAO it happened again below. Apparently I was accidentally hitting Cmd+Enter and that just auto-posts whatever you have)

giogadi fucked around with this message at 16:16 on May 14, 2024

# ? May 14, 2024 16:05

giogadi: Oct 27, 2009

I've seen the template-based pattern you mention, but boilerplate isn't the only problem I'm trying to solve - I'm trying to make it easy to write a for-loop over all my properties (for diffing), which is why I'm going with the array-of-property-structs thing. (I'm aware that you can use template metaprogramming with a kinda recursion-like pattern to do "loops" in templates, but I really don't want to go that route)

I was hoping I could get around the heterogeneous type issue by just casting the member pointer into a pure size_t offset; so for example:

code:

struct Property {
    char const* name;
    PropertySpec* spec; 
    size_t offset;
};

struct FooEntity {
        virtual SomeVirtualFunction();

	int hp;
	float maxSpeed;
};

Property gFooEntityProperties[] = {
	{
		.name = "hp",
		.spec = &gPropertySpecInt,
		.offset = (size_t) &FooEntity::hp
	},
	{
		.name = "maxSpeed",
		.spec = &gPropertySpecFloat,
		.offset = (size_t) &FooEntity::maxSpeed
	}
}

// Then, you can access "hp" through a void* by doing:
FooEntity foo;
void* object = &foo;
Property* hpProperty = gFooEntityProperties[0];
int* hp = (int*) ((char*)object + hpProperty->offset);

My question is whether the above is "safe" - I don't see why it wouldn't be, but maybe there are things going on under-the-hood of classes with virtual functions that I don't understand.

giogadi fucked around with this message at 16:15 on May 14, 2024

# ? May 14, 2024 16:10

giogadi: Oct 27, 2009

korora posted:

code:

template <typename S>
void serialize(S serializer) {
    serializer(�hp�, hp, �maxSpeed�, maxSpeed);
}

Actually, one question about the above pattern: for the functors you pass into this, do you require the functors to be variadic? I can't see how else this would work...

# ? May 14, 2024 16:21

korora: Sep 3, 2011

giogadi posted:

Actually, one question about the above pattern: for the functors you pass into this, do you require the functors to be variadic? I can't see how else this would work...

Yes, but if you don�t want to implement a variadic template operator() in your functor you could split it into N calls to the functor instead.

Not totally clear on the diffing use case but perhaps that is also supported by writing a different functor?

# ? May 14, 2024 16:36

rjmccall: Sep 7, 2007; no worries friend; Fun Shoe

If you�re generating code anyway, just generate the obvious code instead of tying yourself into knots with multiple levels of abstraction.

# ? May 14, 2024 16:38

giogadi: Oct 27, 2009

rjmccall posted:

If you�re generating code anyway, just generate the obvious code instead of tying yourself into knots with multiple levels of abstraction.

I've tried the codegen route as well. It does work but it's exactly annoying enough that I get lazy about using it. Maybe my head is really far up my rear end already but I don't find the above scheme confusing at all; I just want to know whether it's safe to assume that casting a pointer-to-member-variable to a size_t is a safe thing to depend on.

edit: in case it's not clear, I'm hoping to _not_ generate code here. Adding the array of Property structs at the end of a struct declaration would be a manual process.

giogadi fucked around with this message at 16:45 on May 14, 2024

# ? May 14, 2024 16:43

OddObserver: Apr 3, 2009

giogadi posted:

I've tried the codegen route as well. It does work but it's exactly annoying enough that I get lazy about using it. Maybe my head is really far up my rear end already but I don't find the above scheme confusing at all; I just want to know whether it's safe to assume that casting a pointer-to-member-variable to a size_t is a safe thing to depend on.

Pretty sure it won't build, but it's been ages since I used those. I would just have:

code:

class Serializable {
 virtual void Save(blah)
  virtual void Load(blah)
};

template<typename T> class Prop: public Serializable {}
struct FooEntity {
 Prop<int> hp;
 Prop<float> whatever;
};

...Though this wastes memory.

[/code]

# ? May 14, 2024 16:54

giogadi: Oct 27, 2009

Thanks for the suggestions, y'all!

I think the main use case for this weird poo poo I'm doing still isn't clear, so let me try to explain:

In my game editor, I'd like to be able to select multiple entities of the same type and then edit all their properties simultaneously. Ideally, I would avoid implementing this multi-edit functionality separately for each individual entity type. The most straightforward way I could think of to do this was for each serializable struct to have a literal array of Property structs so I could implement the multi-edit functionality like this:

code:

void MultiEditImGui(Property* properties, size_t numProperties, void** entities, size_t numEntities) {
    for (int propIx = 0; propIx < numProperties; ++propIx) {
        for (int entityIx; entityIx < numEntities; ++entityIx) {
            // Interesting multi-edit logic that includes diffing and applying
            // one ImGui element's result to all the entities
        }
    }
}

So my interest is less about avoiding boilerplate and more about being able to reason about a struct's serializable fields as an array to be looped over for the above use case.

# ? May 14, 2024 17:13

rjmccall: Sep 7, 2007; no worries friend; Fun Shoe

If you�re willing to write unportable code that technically has UB, sure, go at it.

# ? May 14, 2024 17:23

giogadi: Oct 27, 2009

rjmccall posted:

If you�re willing to write unportable code that technically has UB, sure, go at it.

Ok, so casting a pointer-to-member as a size_t is unportable and UB? That's a helpful response, thank you.

I've been meaning to rewrite my entity system to use only C-style simple structs, and this is just another reason to throw on the pile.

# ? May 14, 2024 17:27

rjmccall: Sep 7, 2007; no worries friend; Fun Shoe

Yeah, there�s no supported way to turn a member pointer into an offset. Storing it into memory and reinterpreting is mostly portable in practice as long as you aren�t tripping one of the cases where MSVC uses wider member pointers, which IIRC requires virtual bases or incomplete types. But that kind of bitwise punning is pretty much always going to be UB one way or the other.

# ? May 14, 2024 17:39

giogadi: Oct 27, 2009

Just in case anyone's curious, I sketched out a terrible thing using templates that does the multi-ImGui functionality I'm looking for (inspired by korora's suggestion). The main downsides of this approach are:

(1) :barf:

(2) It requires knowing at compile-time what type of entities are in the list. However, in general I would have a list of entities of various types, which I would check at runtime for whether they are all the same and then use the multi-ImGui functionality if so. I can still make this work by having a big switch statement on the runtime-determined entity type that then dispatches to the appropriate type's Serialize() function.

I don't think I'll actually use this, but it was educational to write it out.

code:

// Used to get the underlying type of a pointer-to-member
template <class C, typename T>
T getPointerType(T C::*v);

// returns true if it was changed
bool ImGui(char const* name, int* v) { /*snip*/ }
bool ImGui(char const* name, float* v) { /*snip*/ }

template<typename EntityType>
struct MultiImGuiFunctor {
    EntityType* entities;
    size_t count;

    // For each (name, pointer-to-member) pair, we do one ImGui() element
    // with a value seeded on the first entity's value, and if that element
    // actually gets changed, we propagate that change to all the entities.
    template<typename PointerToMemberType, typename... TailTypes>
    void operator()(char const* name, PointerToMemberType pMember, TailTypes... tails) {
        decltype(getPointerType(pMember)) value;
        EntityType& firstEntity = entities[0];
        value = firstEntity.*pMember;
        bool changed = ImGui(name, &value);
        if (changed) {
            for (int entityIx = 0; entityIx < count; ++entityIx) {
                EntityType& e = entities[entityIx];
                e.*pMember = value;
            }
        }
        this->operator()(tails...);
    }

    void operator()() {
        return;
    }
};

struct FooEntity {
    int x;
    float y;
};

template<typename Serializer>
void FooSerialize(Serializer s) {
    s("x", &FooEntity::x, "y", &FooEntity::y);
}

int main() {
    FooEntity entities[10];
    MultiImGuiFunctor<FooEntity> multi;
    multi.entities = entities;
    multi.count = 10;
    FooSerialize(multi);
    
    return 0;
}

# ? May 14, 2024 18:32

giogadi: Oct 27, 2009

rjmccall posted:

Yeah, there�s no supported way to turn a member pointer into an offset. Storing it into memory and reinterpreting is mostly portable in practice as long as you aren�t tripping one of the cases where MSVC uses wider member pointers, which IIRC requires virtual bases or incomplete types. But that kind of bitwise punning is pretty much always going to be UB one way or the other.

Thanks! If I only use simple C structs, is it guaranteed to be safe to access members through offsetof() like below?

code:

struct Foo {
    int x;
    float y;
};

Foo foo;
void* pFoo = &foo;
float* value = (float*)((char*)pFoo + offsetof(Foo, y));
*value = 1.0f;

giogadi fucked around with this message at 19:01 on May 14, 2024

# ? May 14, 2024 18:58

rjmccall: Sep 7, 2007; no worries friend; Fun Shoe

Yeah, absolutely, as long as the access is the right type for the field.

# ? May 16, 2024 02:16

StumblyWumbly: Sep 12, 2007; Batmanticore!

For fun, I'm working on a desktop program to load a multi-GB file of time series data as quickly as possible while also making the data available while the file is being read. I mostly do embedded, and my thought is to have one thread to read the data into shared buffers, and other threads to process the data as it becomes available, so we're reading and processing at once instead of blocking on the read.

This seems straight forward and obvious to me, but I don't see much discussion of it or libraries that support it. Is it just too much work for most stuff, or flawed in some way?

# ? May 21, 2024 01:53

Sweeper: Nov 29, 2007; The Joe Buck of Posting; Dinosaur Gum

StumblyWumbly posted:

For fun, I'm working on a desktop program to load a multi-GB file of time series data as quickly as possible while also making the data available while the file is being read. I mostly do embedded, and my thought is to have one thread to read the data into shared buffers, and other threads to process the data as it becomes available, so we're reading and processing at once instead of blocking on the read.

This seems straight forward and obvious to me, but I don't see much discussion of it or libraries that support it. Is it just too much work for most stuff, or flawed in some way?

What is as quickly as possible? That�s not a real requirement. What kind of hard drive is it? How long does it take to decode the data? Is the data streamable? Can you parse it in chunks in parallel? nvme drives are insanely fast and the OS is pretty good at prefetching, have you tried reading the file serially? Is this on Linux? Windows? OSX?

# ? May 21, 2024 01:58

Jabor: Jul 16, 2010; #1 Loser at SpaceChem

Your "read into a shared buffer" can just be a call to mmap, at which point you can start processing that data concurrently with the OS actually loading it into memory underneath you. And you don't need to write your processing code in any special way to wait for the next bit of data to become available, you just get that for free.

# ? May 21, 2024 01:59

more falafel please: Feb 26, 2005; forums poster

Jabor posted:

Your "read into a shared buffer" can just be a call to mmap, at which point you can start processing that data concurrently with the OS actually loading it into memory underneath you. And you don't need to write your processing code in any special way to wait for the next bit of data to become available, you just get that for free.

Yeah, this is exactly the use case for mmap. Having a kernel does sometimes have its advantages.

# ? May 21, 2024 02:20

StumblyWumbly: Sep 12, 2007; Batmanticore!

Oof, mmap looks perfect except I do want to run on Windows so I'll look into MapViewOfFile stuff.

Sweeper posted:

What is as quickly as possible? That�s not a real requirement. What kind of hard drive is it? How long does it take to decode the data? Is the data streamable? Can you parse it in chunks in parallel? nvme drives are insanely fast and the OS is pretty good at prefetching, have you tried reading the file serially? Is this on Linux? Windows? OSX?

This is a for fun project so I don't have hard requirements, I'll be benchmarking on my Windows computer with an old SSD, but I mainly want to play around with situations where data retrieval is the limiting factor. The data is streamable and can be parsed in chunks. I plan to start with interleaved X, Y, Z time series data and try making it more complicated from there, like having a header and occasional blocks that require different parsing.

# ? May 21, 2024 03:09

nielsm: Jun 1, 2009

You definitely want to make sure your data format is suited for memory mapping, at least. If it's a CSV file it might be hard to decently seek to a specific record, but if you have a guarantee that the file is sorted by timestamp then it probably can be made work. Ideally your data would have a fixed record size in bytes, so you can treat the entire file as a giant array of structs, but in that case you'd also need to be careful about data alignment inside the file so you don't end up with all your reads unaligned.
If your input data doesn't have any of those qualities innately, then it might be worth building an index into the data as a first pass before handing things off to the user, possibly in a background thread. The index could also be stored as a file that can be memory mapped.

# ? May 21, 2024 03:30

Adbot: ADBOT LOVES YOU

# ? May 25, 2024 00:03

Twerk from Home: Jan 17, 2009; This avatar brought to you by the 'save our dead gay forums' foundation.

That that's the kind of thing I do and see all the time in scientific software, where you've got data that's compressed and parseable as distinct records that they will somehow be individually processed on a threadpool of workers before some kind of reduce or join at the end that's on another thread or threadpool.

What exactly are you looking for libraries to do? The naive implementation, which is usually good enough, is to let the parser allocate and pass unique_ptrs to the threadpool doing processing, or shared_ptr if you really must. Assuming that parsing has to be done in-order and depends on previous state, this still may not scale up to many cores and bottlenecks on your decompress/parse thread, or one of the two if you split those into two different threads. You could clearly optimize with an object pool instead of having the parser allocate if you wanted to.

If your main complaint is that the standard library doesn't have a concurrent, blocking queue of limited size to enable this, I think that's because most situations either a std::queue (or probably std::deque) protected by a mutex is good enough if you have low contention, and if you have high contention you need something industrial strength like https://github.com/cameron314/concurrentqueue or https://github.com/cameron314/readerwriterqueue.

I wouldn't use the lock-free ones unless you really need it, personally.

Edit: Here's the kind of thing you'd do to protect a queue with a mutex: https://morestina.net/blog/1400/minimalistic-blocking-bounded-queue-for-c. Yeah, I wish that this was in the standard lib too.

Twerk from Home fucked around with this message at 03:58 on May 21, 2024

# ? May 21, 2024 03:49

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > C/C++ Programming Questions Not Worth Their Own Thread

«‹›640 »