In dev log #3 we spoke about converging our design efforts into a cohesive whole as we approach v1.0.0. This time we take a look a little past 1.0, and the choices made that will affect our trajectory.

A lot of decisions go into a game engine that seem inconsequential but end up stacking into many divergent paths, but since our ultimate goal is shipping games using the engine and not the infinite engine treadmill - some of the choices have a longer term effect. Let's consider a few!

topsoil

I'm always happy when people ship games with luxe (even if it's in alpha),
so I wanted to take a quick detour to mention this game that's releasing soon.

It's a delightful game and it's coming to android and iOS on April 26th.
The game has a website with all the details and you can retweet this if you're inclined!

on balancing trajectory

Balance is pretty key in every regard, like it's important to balance costs of decisions against their benefits and things like that, but in particular it's quite useful to balance time and convenience.

A phrase I came up with to articulate this is along the lines of "a side effect of timing". When it comes to large projects - like a game engine - time is an important factor since technology changes, developers and platforms evolve, your actual needs may change. They also take a long time to make and even more time to make them work really well.

The state of any aspect of a project at any particular point is a side effect of time/timing, and more specifically, where you are on the road to your actual goals.

This is an obvious statement, but like with the other posts in this dev log, the goal is to give you clearer understanding of my intent and the project as a whole. There are going to be things that are measured on a longer timeline and things that are measured on a shorter timeline. Because there are so many things to consider for a game engine, they may all be in various states at one time.

Concretely, this means some things may be "less than ideal" now, but they approach their ideal goal over time. A good example would be great samples; ideally we want the best samples now already, but as long as the ones that are there are usable and there is enough to go on we can balance when that aspect will reach it's peak (and spend the time finishing the engine).

I put "less than ideal" in quotes because everyone differs (some people like to use a full IDE but some prefer a decent text editor). To me, the balance I'm talking about is whether or not the state as it stands is actually decent and solves what it sets out to solve. Or more specifically, if the state now isn't the full vision but a portion of it then that's ok. If I can accomplish what I need to do and it isn't the worst ever, that's plenty already.

convenience

The best way to view this is the idea of stepping stones. The current state is a stepping stone to the ideal version, but every state should at minimum be as usable/stable/reliable as possible. It's not an excuse for a bad time, but rather a pragmatic approach to developing something lasting.

You can clearly see this trend in luxe if you've been around a while. This is the nature of iteratively developing something, and it works great for me. It does mean that as a user my expectations must face the pragmatic nature of the project, but typically this translates to doing a little more work for my needs, or in other words: convenience.

I like to think that the engine is still nice to use even when in active development, even if some things are inconvenient. Over time however, the timelines will approach their ideal and more and more things align, and the more we work together the more ground would be covered sooner.

finishing

This leads to my point, that as we approach 1.0 the definition of finished is based on the goals of the engine (and not on an idealistic view of it).

The many timelines that will meet at 1.0 are all related to the core and core design goals and they're meeting up now. As a stepping stone above the current state, a large percent of everything (from the user workflow to the systems and tools available) are a significant step above what you're using now.

trajectory

On to the point!

Sitting down to finalize 1.0 had me question what it looks like with the design convergence. I look backward from around 1.0, and ask what the user does day to day. How I want to work, what tools I need, how I imagine those working together, how new or existing platforms are handled, many many things get clearer when considering the full picture. This leads to decisions that affect the trajectory long term.

Some choices are a bet on the future of a platform, some require careful consideration and testing before speaking about them, and all of them revolve around consistency/reliability/predictability of the foundation below the engine - since our goal here is to ship games using the engine, not infinite engine treadmills.

Below I wanted to talk about some of these choices that I made when I sat down to finalize 1.0, and how they relate to the engine and it's vision as we get close to 1.0.

rendering

The trajectory of the renderer was to be aligned with modern API's in the low level, allowing us to leverage platforms like Vulkan and Metal and ES 3 on mobile and what not.

It's designed to facilitate 2D games really well, but also bigger games, but it must do so without those getting in each others way. This took a lot of careful thinking but I am very happy with how the rendering is working now, and I think lots of users will be too.

Many of the design choices I had made with the renderer were somewhat vague originally (a renderer is lot to fit in your head when you're designing it). So now that we're closer to 1.0 I needed to start stretching it's legs a little - making sure that the fundamentals and design goals are being met. I needed to use it to achieve things in practice, which meant implementing some rendering!

Please, keep in mind the previous dev log details about modules and rendering with what you see here. I will go into much more detail about rendering in the future but for sure don't assume that "this is what luxe looks like" or anything - It's just a shader and some render passes! The engine is still very much specifically designed with 2D games in mind, please don't worry about 2D being left behind.

In fact it's significantly better now for 2D, since post processing in general is really fluid, and things like rendering passes easily allow you to configure the way your game renders to the letter (without necessarily having any deep knowledge about rendering). There's also culling and various other things that benefit 2D well, and modules will be able to provide a lot more for 2D rendering in general. Constructing visually advanced 2D games no longer requires several hoops or plumbing to get right - something I know users have had some pain with in alpha.

renderer vs rendering pipelines
The distinction that matters is this: the renderer vs a rendering pipeline. A renderer is the API and systems in place that you use to render things, while a rendering pipeline is what makes up a frame in your game. You can implement several rendering pipelines in the same renderer.

The job of the luxe API is to let you (or a module) decide what happens in a frame, facilitating that in an easy to use but flexible and powerful way. I'll talk about the rendering pipelines later but obviously being able to implement all of this in a few hours of work (for someone experienced with rendering!) means that the engine design is clicking into place really nicely.

The intended purpose of the images below is testing using the renderer, and the module will be available (I'll talk about it in future).

The images below test many explicit things:
(don't worry if you don't know about these things!)

• Rendering passes very easily
• Passes give you post processing, but also shadows, blending, etc
• Rendering depth only pre-pass easily
• HDR rendering with tonemapping + gamma correctness
• Rendering to various texture formats (like packed 32 bit float formats)
• Loading and using cubemaps, textures with mip levels, HDR formats
• Easy control of the rendering from the game/settings/configs
• Portable rendering (i.e adaptive quality, different rendering on mobile)
• Custom mesh rendering (attributes + inputs)
• Dynamic rendering (i.e changing the render path dynamically)

By implementing HDR rendering, fog, bloom, a nice surface material, shadows and more that isn't shown I am a lot more confident that the design is working, and I ran into several issues doing so (which are mostly resolved now). For 1.0, the renderer should be quite formidable in practice which makes me really happy.

One of the nicer things is that general users who don't need to understand all that stuff can still benefit from it, but since luxe is interested in people understanding the things they use to make their games - I'll continue to post details about all that in future.

To avoid confusion with many images, you can view more of the rendering I implemented to test in this thread.

portability

I've been doing native stuff for a long time, so porting to new targets is in my blood stream. If the code is designed well it's fairly easy to port, and I consider luxe already in this description since it runs on 4 native platforms with ease and adding more is no trouble - but it's not where it should be for 1.0 on the system level.

The previous iteration of the engine has some flawed choices (I mentioned some in the convergence post) which get in the way of this though, a good example is the renderer not being an abstraction, it used OpenGL directly, which means that porting to any targets without OpenGL can be problematic. The right way is a proper backend, where your backend just implements the user facing API instead.

C++ runtime
Another important factor that gets in the way of native targets is porting (and maintaining) the many moving parts. Especially in building and running the engine, it's critical for the engine to be able to adapt quickly. It's especially important for it to be stable and maintainable by me (and any devs that make in house modifications to the engine). High complexity in this regard would be a detrimental choice to the engine long term, one I take pretty seriously.

For the longest amount of years it is undeniable that the most portable option available is C, it's sort of the baseline for what portable actually means on native targets for most of us. It interacts with almost every language out there and it's a well defined space. Since I don't really want to program in pure C at this point, C++ with a C API is the next best thing for this goal, giving the most direct path to any native target that the engine cares about.

This also works well for complimenting the engine with other languages, and as mentioned in dev log 3 bringing back the high level scripting as a first class workflow - the user doesn't have to or shouldn't have to deal with c++ to make their game if they don't want to. That includes i.e ever having to install visual studio or fight c++ errors when you're just trying to make a game.

performance
One design goal I care a lot about relates to performance. For 1.0, I wanted to ensure that the runtime performance is able to carry the weight of active development of a moderately sized game (the ones I make) with relative ease as far as performance goes. In other words, if I'm developing some AI system for my game and I want to visualize a lot of the debug information on screen, having the engine running at 15fps makes it difficult to work.

The engine should be able to handle messy, rapid iteration without it breaking workflow. It should give you the slack to build, not just to have an optimized final product, but acceptable workflow when you're iterating day to day. In the worst case, having to constantly address performance issues during active development isn't fun for anybody.

I had all along intended to address this by taking the core systems to the metal so that as a developer you can worry about the game part.

parallelism
The engine also has to account for the fact that it's 2017 and my phone in my pocket has 6 CPU cores waiting around doing nothing (if the engine is single threaded). For 1.0 the engine is designed to be able to work in parallel, so that it can more effectively use the available hardware.

Since this involves a lot of consideration the design is built for it, but since I want to get 1.0 into your hands sooner than later, it may not be utilized to the fullest extent when we're at 1.0. This is a perfect example of the trajectory, I know where the engine needs to be, design and implement for it, but it may arrive after the 1.0 timeline.

good enough
The metric used for performance is quite important because it's an ongoing thing to tackle and is hardware/platform specific. luxe uses "good enough" as the metric, in the sense that like above, it will carry the development costs if it's able to. It will do what it can to utilize the hardware available (like multi core and SIMD optimizations where it makes sense). But there's a definite rabbit hole here that I use good enough to avoid.

A concrete example: if the animation system can update like whatever 50k animations in less than 1ms that's plenty fine and I'm not going to over complicate the code or api or engine design in order to get faster all the time. It's good enough, and will continuously improve without sacrificing other core design goals (like low complexity and maintainability). Optimization for the sake of it leads to generic solutions, solutions to problems that don't exist in any project - and I'm not a fan of those.

conclusion: native targets run on C++ with a C api

So this decision was pretty easy to make, and I made it very early when I sat down to finalize 1.0. It's a return to the original design of the engine (before it was called luxe) and I am very happy with where it's landed. I am confident you will too when I expand on more details of how this looks.

web target

The above raises an interesting question about the web target, how do I want to handle it? Web target is pretty important to me (and to luxe) but web has some pretty frustrating parity with desktop in some ways. In particular with rendering, and in some ways with performance. There were two new emerging technologies for web that I've kept a close eye on since their inception: WebGL 2 and Web Assembly.

I had bet/hoped that they'd be ready by the time I had to answer this question, which would open up a more straight forward path to a really great web build, if the timing worked out. The great thing is that it has, and we're able to target web directly from the same C++ code in stride, which gives us immediate access to both of those things.

emscripten

Anyone familiar with web targets should know about emscripten by now. If you don't know of it, it allows you to compile C++ code to an optimized subset of javascript which browsers support, it's a compiler toolchain.

I've revisited it a number of times in the past few years, but every time I did there was some thing I didn't exactly find ideal. I knew that if i was going to consider it for luxe it has to have ironed out those issues by the time I get there. Things like setting it up to compile, documentation about the details and complexity involved, consistent browser support, actual case studies by other developers (preferably larger ones), API support for browsers, build sizes and things like that.

emscripten is really great now
My experience this time however has been pretty flawless, so respect to everyone involved in that project because it was a mile and a half better to use!

From setting up to compiling the entire luxe code base took around 15 minutes - no kidding. I had some valid code built out in no time, and the remaining time to get the build up and running was fixing assumptions made earlier on the engine side (for the sake of moving fast). All in all, I had the engine running smoothly on web in a matter of hours. This is on par with other platforms (like iOS/Android/linux/windows/mac), so I am glad.

It also uses the same build workflow as all the other targets, and there are a lot of benefits that fall out of using emscripten.

Code parity
One of the biggest wins is shared code. Since the engine runs on SDL2 by default, emscripten actually has a whole SDL2 port available to use without any setup. I added one build flag, and the same code that launches the engine on all native platforms was now giving us a window on web.

The other parity comes from things like OpenGL ES 2 and ES 3 support in the engine renderer backend. In browsers, WebGL 1 is very much just ES 2 level API's with minor differences, and WebGL 2 is pretty much ES 3 as well. This is great because in the backend, if you're using ES3 API's, your "window" in the browser will be running on WebGL 2, and the code is the same as the ES 3 code that runs on iOS or Android. This saves a lot of potential bug hunting and disparity between platforms.

API access
It also has great access (from your C++ code) to the web api's, like fullscreen/gamepad/pointerlock/webgl/css and so on. Even though 99% of that is handled by the shared SDL code, it's still great that it's an option to expose as needed.

Web Assembly
Web Assembly is exactly what it sounds like. It's an assembly level language designed for the browsers, which gives close to native performance (actually) and is portable, small, and more optimized in many ways than javascript.

Compiling to web assembly was another build flag added to emscripten and hey look, it's web assembly! It is new but it's already reached cross browser consensus and enabled by default in modern browsers. We can still target js builds of course and support a wide range of users but luxe is built for long term trajectory, and this is an excellent place to be right as it's ready.

WebGL 2
WebGL 2 also recently shipped cross browser on by default, and it is really nice if you want to target any rendering this side of 2010. There are so SO many huge improvements to WebGL in this version that reduce code disparity and align significantly closer to all other platforms that I am very happy it's ready.

I'll go into detail how that fits in in the future, but the engine rendering abstraction not having to target a many-year-wide gap for rendering API's is obviously a win for anyone shipping a game.

Since luxe is built against it's future, the new renderer is designed on modern rendering API's, and WebGL 2 compliments that well. This doesn't mean current users are ignored of course! But it does mean as time moves forward 1) we're ready 2) we don't have to compromise the engine design because of the past.

Build size
One concern I had originally was build size (because giant builds can be painful). In practice though the engine code has compiled down to around 400kb wasm, and around ~680kb for the js build (after being served over gzip, which should be your default for games). That's a small difference between the hand written version, but you won't get web assembly performance from the hand written version regardless so I'll gladly pay the few extra kb for the code parity and performance at this stage.

I'm sure as tools for web assembly mature this will come down too, but build size still being under 1mb is good enough for me.

conclusion: web target uses emscripten
Works right now, runs existing target level (WebGL 1) and aligns well with the future (Web Assembly + WebGL 2). Build size is decent, performance is great, balances well for the engine needs immediate and long term. Shares code + build workflow with all other platforms.

Another comparatively easy choice!

Other targets

I mentioned previously that I was focused on finishing the core systems and avoiding porting to other targets gives me the space to focus. As things get closer to 1.0 now it's easier to reliably add and test the other targets again. Since it's not just whether the code runs there, but whether the workflow is usable as well.

One thing that I do when porting initially is removing unknowns (like maybe some issue comes up with alignment of data written by the asset system) and once the target is set up, it's no longer a concern. So web is no longer a concern, what else have I been up to?

Recently I got the new iOS target running, which is running great. Windows is running great, Mac is running great (it's my dev os) and that just leaves Android and Linux but those are both quick to add (as mentioned, a few hours usually at most).

We'll soon have all the targets running smoothly and all the systems wired up and 1.0 follows right behind that.

We also have some new native targets we're working on to talk about but unfortunately it's too early to say anything about them. Keep reading the dev logs though!

Next time...

So if we have our targets lined up, our goals are aligning really well and things are running smoothly, there are still a number of questions that come up with regards to portability. Since this post is already decently long, I'll have to discuss more of those in the next post.

If we can target WebGL 1 & 2, OpenGL ES 2 & 3, and desktop OpenGL 2/3/4 easily with a nice rendering abstraction how do we deal with shaders being portable?

This is another long term choice which has timeline considerations, usable now, but even better in the long run. The answer is a brand new shading language written for luxe, and I'm excited to talk about it. Still plenty of work ahead but we're marching closer!

Until next time!

You can follow news about luxe on twitter @luxeengine or by subscribing to the feed or the mailing list above.

Feedback on this post can be found below or shared in this discussion.