Yes, and more. You’ve also got a rigid body (physics) simulation, ai/pathfinding, animation, networking, audio and gameplay logic all to run. If you’re making an open world game, you also need to have streaming code to load and unload parts of the game on the fly.
It doesn’t always happen at 60hz (which is 16ms total processing time for all the above plus what’s in the post). Some games run at 30hz(33ms), some run unlocked (varying times), and some can even run at 144Hz (7ms).
As an extreme example, VR headsets normally require rendering twice (once for each eye), and run at 90FPS, which is 11ms to do your entire game, and render it twice.
Furthermore, these aren’t normally just guidelines, these are caps. If your game runs at 60hz, and you miss your target by 1ms, on most hardware you’ll end up missing the hardware refresh which means either you get a temporary drop to 30hz, or you get tearing (half the old image and half the new image), neither of which are particularly pleasant.
To clarify - In many cases you are actually running your physics or game loop at a different rate from the render loop. For instance, in CS:GO you can have a frame rate of well beyond 300, but the game loop (physics/ai/net/logic/etc.) is ticking at 60 (by default) due to the client-server architecture. This type of non-synchronous engine architecture is very complex to build reliably, so unless there is a hard requirement for it (e.g. client-server multiplayer model w/ advanced latency compensation) you will usually find a simpler synchronous approach used unless the underlying engine comes with the async architecture OOTB (UE4).
That's not exactly actually (UE VR dev here). When you render a game at 90hz (or 60 or really anything), that means your total throughput needs to be 90hz, NOT that the frame needs to render in 11ms.
Since you're dealing with multithreaded CPUs + an asynchronous GPU, you can parallelize and sequentialize all this. How UE works in its DX11 and OpenGL renderers, on a 90hz game, is that you're gonna have the Game Thread (Physics/Gameplay) running for frame N, while the Render Thread (GPU commands / math ) runs for frame N-1, while the GPU executes frame N-2. This allows you a complete frame time of 33ms on a throughput of 90hz, at the cost of more latency.
At some point the perception of the human eye means you cannot increase latency further. Especially if the game is something where reaction time is important, like a twitchy first person shooter.
I wonder what the cutoff is where someone starts to notice, but I imagine it's not much greater than 50-100ms
Minor nit: some of those needn't run every frame (ai/pathfinding, gameplay logic). It's also possible to calculate future points and interpolate between them per frame. A nasty trick is to have shadows running at a half framerate (Crysis did this).
I'm curious how you would relate in-browser WebGL performance vs. performance of native software such as Unreal. Is running it in a browser 10x worse? or 100x?
It's not going to be a simple constant factor. The ideal case will be the same - ultimately the CPU is doing the same work (so provided the JIT picks it up correctly it'll be executing the same instructions) and the GPU is running literally the same shaders, so performance should be identical. It's more a question of how much work you have to do to hit that happy path, and what edge cases pull you off it.
The CPU is going to do some more work because WebGL can't allow the GL app to crash the machine or break into the OS kernel, which regrettably current OpenGL (And DirectX, And Metal, and Vulkan...) drivers allow.
Allowing a userspace application to crash the machine or break into the OS kernel is also a security violation that needs to be prevented; the consequences are less severe than when a web page does it, but it still shouldn't happen. So that should also be the same work in either case.
In principle WebGL should be competitive with OpenGL. Once it hits the GPU, it's the same.
Some of the bottlenecks in WebGL are:
- Javascript. Modern JS engines are great, so the overall speed can be good, but you're still missing things like 64-bit ints, SIMD and threads. (Do game engines make heavy use of threads though? I don't know, but many influential games programmers seem to be wary of them!)
- The WebGL -> OpenGL translation layer takes some time. In Chrome it sanity checks your input (which you definitely want in a browser! GPU drivers are very insecure) and executes it in a separate process. Not as expensive as you might think, especially if you minimize your draw calls, which is good practice anyway.
- WebGL is basically OpenGL ES 2.0 (and WebGL 2 is ES 3.0), which is missing some useful features of full OpenGL, and doesn't offer the low-level, low-overhead access of Vulkan or Metal.
Depending on what you're doing, I'd guess WebGL might be no more than 2x slower (assuming plenty of optimization work). Some fiddly things might be 10x slower, or just not possible at all within the WebGL API.
WebGL 2 has a lot of very important new features, but support for that still seems to be patchy, and it's only just catching up with mainstream mobile graphics. It's a generation behind Vulkan.
Apart from that, an AAA game will have massive amounts of graphical and audio assets. Delivering that over the network is a pain and HTML5 caching is a pain. Doable, but hardly comparable to just loading it from local storage.
Oh, and one more thing! A big game wants sound as well as graphics, and WebAudio is a mess. And audio mixing is typically done in a background thread, so that's one area where the lack of threads in JS is a real problem.
Overall WebGL is very nice, it was a great choice to follow OpenGL ES closely (security problems aside).
Can you go into more detail? (Or point me at some up-to-date books or blog posts!)
I'm specifically curious about CPU-intensive stuff that is sharded out to multiple threads. That's your classic multithreaded programming, the sort of thing you'd do in scientific computing, but I always got the impression games people were skeptical about it, due to unpredictable performance and the high risk of bugs.
Motivation: The Xbox360 pretty much forced gamedevs into heavy threading if they wanted to get anything done. It had 3 PowerPC cores with 2 hardware threads each. The cores had huge memory latency and no out-of-order execution. IBM's attitude about OOE was "Statically compile for a fixed target" and "Run 2 threads per core and that that'll cut the effective stall cycles per thread in half". The PS3 had only 1 of those PC cores, but it also had 6 unique cores that were practically high-power DSPs. If you manually pipelined data movement and vectorized execution, you could get amazing results. If you ignored those cores, the PS3 was crippled. The XBone and PS4 have friendlier cores, but they are still surprisingly low-power and there are 8 of them. So, you still need to thread and vectorize or you'll be dragging. Even on the PC, Sutter's "The Free Lunch is Over" is over 12 years old. Outside of games, the browsers force single-threading and cloud servers profit by selling multicore machines pretending to be many single-core machines. But, in games have to run on non-virtual hardware.
Execution: You can google around for "game engine job system", but unfortunately, game engine blogs have really fallen off a while back as most of that crew has moved to twitter. So, the best material out there is in the form of GDC presentations such as "Parallelizing the Naughty Dog engine using fibers", " Destiny's Multithreaded Rendering Architecture", "Multithreading the Entire Destiny Engine", "Killzone Shadow Fall: Threading the Entity Update on PS4". Slides and videos are available around the web.
I never got to program one but I remember being fascinated by the weird architectures of the PS2 and PS3.
It occurs to me that there are some similarly weird architectures in mobile right now -- it's not uncommon to see Android flagship phones with 8(!) cores, which is just ludicrous. And there are lots of asymmetric "big.LITTLE" designs with a mix of high-speed and low-power cores.
Maybe I'm just reading the wrong blogs, but I've barely seen any discussion on how to optimize code for those crazy Android multicore CPUs, even though it seems like there's potentially a lot of upside. I guess Android is so fragmented and fast-moving that it's a tougher challenge than optimizing for two or three specific games consoles; also Android app prices are low so there probably isn't as much motivation.
Also, Apple is miles and miles ahead of everybody else in mobile performance, and they've consistently gone with just 2-3 cores. Their mobile CPUs and GPUs are very smartly designed, really well-balanced.
There’s also this talk - http://developer2.download.nvidia.com/assets/gameworks/downl... which talks about collision detection and collision resolution on the GPU that actually explains the architecture of the cpu engine quite well and shows what parts are serial and what parts are well parallelised.
Modern game engines parcel out work via a job system, where functional tasks are dispatched to the cores in a system. This is opposed to the idea of 'one core/thread will own a task for the lifetime of the application, while checking in with a master core/thread'. Actually, most games have a hybrid model. Usually one thread/core is dedicated to rendering, another thread/core is dedicated to high priority tasks/jobs, and then the remaining resources are used by whatever jobs are left.
That threading style potentially fits OK with the JS "Web Worker" / "isolate" model, as long as you can pass messages around between threads/isolates very efficiently (probably not the case in current JS implementations).
Oh interesting. TBH, I have zero knowledge on how to do any of this on the web side. All my experience is from working with traditional game engines on the native side.
The worker pooling system you describe is eminently possible in the browser these days. Web Workers [1] are really just threads with a JS execution context and a facility for messaging back to the thread which created them. (Or, if you set them up with a MessageChannel [2], they can do full-duplex messaging with any thread that gets the other end of the pipe)
Of course, you're still dealing with the event loop in most cases, which is probably a stumbling block when it comes to really low-level stuff. That said, there are even facilities for shared memory and atomics operations [3] these days, which helps. I've messed around with it a little bit on a side project- as a JS developer, it's really weird and fun to say "screw the event loop!" and just enter an endless synchronous loop. :D
Not an expert, but the version of it that I've heard is your main game loop handles all interaction with the game state to avoid issues. You can offload rendering into another thread pretty safely, and anything that alters game state would queue up its state changes for batch processing.
That way if you're running a multithreaded physics simulation you can get two separate bullet collision detections on one object, and instead of trying to delete the object twice you put both "kill this thing" actions on a to-do list. When it comes time to handle that in the main loop, you sweep through it for conflicts before executing any of the changes.
UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that. But here's an example (without much technical detail) of someone implementing multithreaded pathfinding for UE4: https://forums.unrealengine.com/community/work-in-progress/1...
UE4 in particular I know handles all game logic in a single thread, and you can't touch UObjects from outside of that.
Yeah, that's the kind of thing I was thinking of when I said games programmers seemed skeptical of threads.
Some of that coarse-grained parallelism could possibly be done in JS with Web Workers, but those have their own problems. (See the recent discussion here about the "tasklets" proposal: https://news.ycombinator.com/item?id=15511519)
WebGL isn't trying to be competitive with full blown game engines, is it?
For one, game engines normally don't use OpenGL unless they have to. The GL drivers on PC tend to be very weak. High end games target Direct3D on Windows/Xbox and these days are moving to Vulkan/Metal, even on mobile.
On Windows the GL driver situation is so bad that Chrome translates GL to Direct3D. This obviously will impose some overhead and complicate the driver bug situation still further.
Even when games do target GL they tend to exploit lots of vendor specific extensions and be tested against very specific graphics card/driver combos to enable them to workaround bugs and performance cliffs. Does WebGL even expose driver specific extensions? I don't think it does.
So you are not going to be competing with native apps on the web anytime soon in this area (as with all other areas...)
id has always made opengl engines. They get to deal with endless driver breakage. When Rage came out I believe AMD cards didn't work at all for over a week.
Bioshock was a DirectX game and I recall that shadows were broken on AMD at launch. There are some OpenGL-specific problems with driver breakage, but AMD would be in much better shape today if that's all it was.
If you read through the patch notes for your video card drivers, you'll probably notice that every major game release gets special support right in the driver.
NVIDIA has three advantages:
1. Most PC gamers have NVIDIA cards, so developers test primarily against NVIDIA cards.
2. NVIDIA has a boatload of driver developers to hack game-specific fixes and improvements right into their driver. They work around game bugs, rewrite slow shaders, and other stuff like that. I assume that's part of why the graphics driver is now well over a hundred megabytes.
3. NVIDIA sends developers to major game studios to optimize and add graphical effects to their games. For instance, volumetric lighting in Fallout 4 was added by an NVIDIA employee.
I think AMD does much of the same, but they just don't have the money to do it to the same extent. That means more breakage and slower fixes.
WebGL isn't trying to be competitive with full blown game engines, is it?
Sure it is! Not right now maybe, but the web standards people working on it would love to be a viable platform for AAA games. Every so often they tout a new WebGL port of a well-known game as the harbinger of things to come.
Then why are they implementing an API that's been unpopular for years and is now being phased out entirely (for AAA games)? I think they're more concerned with the politics of it than the technical requirements of that userset. Games are heavy users of driver and card specific extensions for instance. But that'd be at odds with the web's portability commitments.
(I'm mostly playing devil's advocate here, I don't actually think HTML will be suitable for high-end games in the near future. But I think there are decent arguments to be made...)
Then why are they implementing an API that's been unpopular for years and is now being phased out entirely (for AAA games)?
Mobile. They picked OpenGL ES 2.0 for WebGL because it had comprehensively won on mobile. Apple went with ES 2.0 and Android followed. It's taking a long time for the mobile industry to migrate to ES 3 (which would allow WebGL 2) but ES 2 has been a decent stable baseline for a good few years now. That's quite unusual, and very helpful, given how fast-moving everything in tech is.
[Edit to add: ES 2 is based on GL 3, which was the first version to add programmable shaders. That was a huge API change, and an admission that the D3D approach was better. It's barely 10 years old. So any "GL is unpopular" arguments based on the old fixed-function pipeline are a red herring, I think.]
Mobile was and is more important than either desktops or consoles, because the mobile market is huge and growing, while desktops and console are at best stable.
Google came up with a clever technical solution (the ANGLE library) to emulate ES 2.0 on top of Direct3D, so that sidesteps the technical problems of OpenGL on Windows.
Now, for AAA games, desktops and consoles are obviously far more important. I think there are two responses to that:
First, a bet that mobile will gradually catch up and become equally important. There are a lot of factors involved, but on raw technical terms it's not such a bad bet. Mobile hardware iterates very fast, and some mobile CPUs are getting very competitive with desktops (recent iPads and iPhones especially). Sustained performance is an issue, as mobile devices have much stricter thermal limits; but you can put the same mobile SoC in a bigger box, like the TV set-top boxes that Google, Apple, Amazon and others are experimenting with.
Second, there's no reason WebGL 3 couldn't be based on Vulkan. WebGL 2 hasn't even been fully adopted yet, so it would obviously take a number of years to make that happen. Maybe desktops and consoles will have moved on to something newer and better by then, but maybe they won't.
The big question is whether mobile+web is catching up on desktop+console, or if it'll always be a generation or two behind. I think you'd have to be pretty brave to bet against them ever catching up.
I think they're more concerned with the politics of it than the technical requirements of that userset.
I'm sure politics plays into it, but for WebGL specifically, it must have been a pretty easy technical decision. Do you pick the 3D standard used on Windows, or do you pick the lower-end one used by iOS and Android (and can be made to work on Windows)?
You could ask why GL rather than D3D won on mobile in the first place. For that you have to look at Microsoft and ask why Windows Mobile failed (in all its different versions). I don't think you can blame that entirely on politics.
Games are heavy users of driver and card specific extensions for instance. But that'd be at odds with the web's portability commitments.
That's a good point. From a web standards standpoint, it's a very tough conflict to resolve. I think the web people are pushing for common standards. That takes time and it can get very political, but I don't see a better solution. And if they can get it right, portability is a good thing! I don't see why that necessarily means you'll always be behind the curve on performance. A more standardized, portable system can catch up via economies of scale -- it might be easier to learn, have better tooling, a bigger potential market, etc.
I think your argument shows why the web is such a poor platform in many areas.
Second, there's no reason WebGL 3 couldn't be based on Vulkan
Vulkan is a very low level API designed for ultra-high performance use in engines written by professional engine teams, like Unreal. It requires the developer to write large quantities of code to even render a single triangle because you have to take manual control over the GPUs low level details. To a large extent it's preferred to GL because of better interaction with multi-threading.
It'd make no technical sense to try and expose a low level hardware-oriented API designed for multi-threading to a slow single-threaded language like JavaScript.
Do you pick the 3D standard used on Windows, or do you pick the lower-end one used by iOS and Android (and can be made to work on Windows)?
Somehow C++ does not have this problem. So how about: don't pick, expose all of them and let the developer use whichever is more appropriate?
You say you don't see any alternative to how WebGL handles driver extensions. Of course there are alternatives: just expose them all. Let there be vendor specific and proprietary stuff in web apps. Just because this is considered politically unacceptable by the ideologues who control the web platform does not mean it's actually unthinkable.
But that'd be against how the web is "designed" (browser makers cabalistically picking winners).
So in fact, I will continue to bet against the web and against mobile. People have been predicting total domination of iPhone/iPad since the day they were launched. I'll still be playing AAA games on consoles or high end Windows PCs 10 years from now, I'm sure of it.
It'd make no technical sense to try and expose a low level hardware-oriented API designed for multi-threading to a slow single-threaded language like JavaScript.
Web Assembly will be mature soon, and on the timescales we’re speculating about it could well have some form of threading.
Let there be vendor specific and proprietary stuff in web apps.
Unfortunately that would be a security nightmare. The Flash and Java plugins are good examples.
The situation has changed a little bit recently with the introduction of adaptive synchronization monitor standards(GSync and Freesync). On PC this means that framerates above some minimum target can drift without experiencing tearing or missed deadlines.
Indeed, that's what I meant by "most hardware", but that hardware is definitely outside the reach of most people right now. Firstly, consoles don't support adaptive sync, and most tv's don't either, so that eliminates all PS4/XB1 (and their derivatives). Then to actually get a screen with GSync, you're talking ~350 pounds [0] for an entry level one, and realistically you're going to need a medium to high-end graphics card (looking on nvidia's website seems their support is actually far better than I expected it to be). Unfortunately, for the majority of people adaptive sync is still a few years off, and I'd be surprised if we saw it this decade.
It doesn’t always happen at 60hz (which is 16ms total processing time for all the above plus what’s in the post). Some games run at 30hz(33ms), some run unlocked (varying times), and some can even run at 144Hz (7ms).
As an extreme example, VR headsets normally require rendering twice (once for each eye), and run at 90FPS, which is 11ms to do your entire game, and render it twice.
Furthermore, these aren’t normally just guidelines, these are caps. If your game runs at 60hz, and you miss your target by 1ms, on most hardware you’ll end up missing the hardware refresh which means either you get a temporary drop to 30hz, or you get tearing (half the old image and half the new image), neither of which are particularly pleasant.