The latency between the GPU and CPU is a very big problem for gaming, if it wasn't we weren't be having this discussion a bad CPU call can easily add 50MS of latency to your frame which makes you all of a sudden drop from 60fps to 20.
You are talking about two different things. One is the latency of data from main memory to the gpu memory and the other is whatever video game metric you are using.
To get 60fps your video card needs to push out a frame every 16ms.
If say 6ms of this is actual in card processing and 10ms is CPU/API overhead and you add to it another CPU call + system memory access that adds 30-35MS latency you are now operating at ~50ms per frame which means that you can only output 20 frames each second.
