> When you have branching code running on a GPU, the threads are split into the branches, and each branch gets executed separately.
I'm not an expert on GPU hardware but in my readings on this, I've seen it said that if there is a branch in your code, all cores take that branch even if it is a no-op for many of them, thus all branches are taken by all cores when the code is not coherent. Hence why making GPU parallelization can be quite a different programming paradigm to really take advantage of how it works, and it is more challenging to do properly than truly independent CPU threads that do not affect each other when they branch.
Ideally all "threads" of the wavefront take the same side of a branch, so it can skip the not-taken side of the branch. Wavefront divergence is when a single wavefront has threads that take different sides of a branch, so the whole wavefront runs both sides and masks out the results based on branch direction per thread.
Thanks for the reply - I hadn't actually thought in any depth about what's going on, and I'd like to get in GPU programming at some stage.
Also, sorry about the barely prompted and completely unjustified wall of rambling text - this is something that I really should've given more consideration to already.
That's a programming model that has the potential to go very wrong, very fast if you don't think in depth about what you're doing.
Branches with branches are going to be very problematic (with exponentially decaying throughput), although multiple branches at the same level are handled very cleanly.
To be honest, I don't know how I thought that it might be rescheduling things instead.
The sort of rescheduling that I seem to have been thinking of, could only make a difference in cases where the batch size exceeds the wave size, and my guess is that the factor of difference would need to be large to make an impact.
At the base case, NOPs and rescheduling would perform identical operations - so all the scheduler would get you is a hardware overhead (the time overhead could be mitigated when running on a single wave).
The scheduler would also introduce latency since different waves in the same batch would need to wait for each other before rescheduling could proceed.
You'd cause problems for your memory layout - programmers referencing values from threads that have branched would probably need to be treated as undefined behaviour.
You'd also need to introduce a stack, per thread, to keep track of the wave histories - allowing you to unsort and re-reschedule at the ends of branches.
All this to run parallel execution on what seems to be uncommonly large batch sizes (I believe AMD just dropped their wave size down from 64 to 32 - I don't know if this is because 32 is a Goldilocks batch size, or if it's simply to achieve better performance on Nvidia optimised applications).
Perhaps this should be threads, but certainly not cores - it's a single core running the same operations on different data across multiple threadish things.
I'm not sure if there's a more technically accurate terminology for quite what they are.
I'm not an expert on GPU hardware but in my readings on this, I've seen it said that if there is a branch in your code, all cores take that branch even if it is a no-op for many of them, thus all branches are taken by all cores when the code is not coherent. Hence why making GPU parallelization can be quite a different programming paradigm to really take advantage of how it works, and it is more challenging to do properly than truly independent CPU threads that do not affect each other when they branch.