AVX instructions (at least the one's we're talking about here) are fully pipelined and none have microcoded implementations that I'm aware of.
And recognize that the news here is "on Haswell". On earlier SSE/AVX implementations POPCNT is indeed faster. Something changed with Haswell that allows this code to run faster than the purpose-designed instruction.
Most likely those two adjacent PSHUFB instructions were limited on earlier architectures to running on a single execution port (with a "PSHUFB engine"), and on Haswell it got insantiated on another ALU and they can now be executed in parallel.
There's no need to speculate, this stuff is all in Intel's manuals. All shuffles are port 5 on Haswell, and they're single-cycle latency / single-cycle throughput. Actually, Nehalem - Ivy Bridge could execute two PSHUFBs per cycle; Haswell reduced the throughput while adding a 256b-wide version, so the total work per cycle remains constant if you adopt the new instruction.
The SSSE3 popcount implementation was never bottlenecked on PSHUFB[1]. The speedup is because Haswell is a physically wider machine (it has more execution ports) and can execute more uops each cycle.
[1] Except on Merom, where PSHUFB was cracked to 4 or 5 uops IIRC, but that's a ten year old part now.
Too late to edit, but I mangled the last sentence of this comment; it should instead be something like "The speedup is because Haswell has wider vector instructions and more execution ports (it can execute more uops each cycle)."
Something changed with Haswell that allows this code to run faster than the purpose-designed instruction.
The change is primarily that AVX2 (which Haswell is the first generation to support) extended binary and integer vector operations to 32B registers, while AVX only supported 32B floating point operations. Instructions that previously operated on 16B doubled their throughput by handling 32B with (usually) the same latency: https://software.intel.com/sites/landingpage/IntrinsicsGuide...
I thought Agner had shown that the Haswell AVX2 stuff didn't necessarily bother turning on the entire execution unit unless it really seemed warranted, preferring instead to issue the 128-bit operation twice and combine. For example see the later comments on http://www.agner.org/optimize/blog/read.php?i=142
There's a real effect here, but in practice I haven't found it to be an issue. As long as your instruction mix has at least 1 256-bit operation in the last few million instructions, the slowdown doesn't happen. I'm sure you could construct a case where it would be a problem, but throwing in an occasional unused VPXOR solves it easily enough.
One thing that can be an issue is alignment. Unless your reads are 32B aligned, you will be limited to reading 40B per cycle. While a single unaligned vector read per cycle isn't a problem, full utilization of the increased throughput that 'stephencanon' mentions in the sibling is only possible if both vectors are 32B aligned: http://www.agner.org/optimize/blog/read.php?i=415#423
The other critical piece was that Haswell doubled load/store throughput to L1 cache, not just register widths. (I know that you know this, just want to make it explicit).
And recognize that the news here is "on Haswell". On earlier SSE/AVX implementations POPCNT is indeed faster. Something changed with Haswell that allows this code to run faster than the purpose-designed instruction.
Most likely those two adjacent PSHUFB instructions were limited on earlier architectures to running on a single execution port (with a "PSHUFB engine"), and on Haswell it got insantiated on another ALU and they can now be executed in parallel.