The first luxe dev post in it's own space!
It seemed fitting to talk a little the design process that is driving the project, and a bit about the road to finishing it.

Design by exploring

People may know by now that I like to work by exploring ideas in practice. This simply entails knowing where you need to end up, but not knowing the exact specific details you need in place to get there. Pick a potentially good direction, and then move in that direction and see what you learn.

To me, code is a fluid tool, a flexible and malleable context in which to express intent and ideas. This often entails throwing together large quantities of code quickly to answer unknowns, and then throwing it out and doing it nicely with the better understanding in place. If you've been around a while, you would have seen me do this multiple times on the repos around luxe and you'll see it again before luxe is finished.

There is a time though when you have to take all the things you uncovered about your design goals and solidify them into code that will stick. This time has been approaching, and what I've been working on.

Logical progressions

In the current iteration of luxe there are a few things in active progression, some more obvious than others. These are the type of things that aren't well defined yet so they haven't been committed to code properly. A good example of this is the rendering or the Entity/Scene relationship, and less obviously would be the draw API.

This is largely because the code that's in there is a placeholder while we explore and validate some questions about the needs of the engine. Not just the hypothetical needs, but the real world ones from making games and shipping them. I know most people seem surprised when I say that a large part of the current code is exactly that, it's just holding a place. Especially during alpha, it's just code thrown in to help us answer the design requirements - They're reasonable defaults that do something obvious and nothing more.

Take the scenes, I know we will need a system to manage a list of entities but what does it look like? I don't know in detail (well, I do know thanks to this approach!), but while we wait: here's a simple container and here is a simple entity. Take the renderer, it's been stated that the renderer is a placeholder and the new one will be designed based on the clear understanding of what it is the engine needs. A lot of stuff in the current code is the type of bare minimum stuff, it's just needed to exist and work enough to help discover the requirements for the key parts of the engine.

The outcome is the point
It may sound concerning, but the key point to note here is that it's a fluid space where we can understand what we need, and then commit it to an iteration that fits well, and grows well. The code along the way isn't representative of the anything but the progression to the end result.

Our goal here, ultimately, is to make the engine according to the vision. This approach is the way I know how to achieve that, and to get pretty precise, so that it ends up being something that I love using and something that stays out of my way. (If you love using the engine already, I'd say it's because I've taken my experience doing this over the years and applied it to the current code).

Where are we now?

This is probably worth asking if I've just said a lot of stuff is placeholder.

It's a good question to keep asking actually, but I feel better about saying this type of stuff now rather than earlier. For me, it's easier to show than to tell, especially if the approach feels unconventional. The above shouldn't be a surprise to the people that are following along with the engine development, since they see the cycles happening in the progression naturally.

Which I mean to say, this isn't new, it's how we got here in the first place and how games have been shipped using the engine during alpha.

core <=> modules
In order to better articulate where we are, it will be easier to start with a short recap of how luxe is designed.

The engine is designed as a facilitator. This means that there is a core of the engine and it serves to provide a consistent foundation upon which to build modules. It's structured in a way that there is a clear line drawn between the core of the engine (the facilities that it provides) and these modules, because the engine should not be concerned with game specifics. It should not attempt in any way, to solve every problem. It has a list of problems to solve, and it must do them well and no more, no less, ideally. This keeps it portable, simple, maintainable (and achievable).

This hasn't been explicitly made clear in the current code for fluidity sake, which just means that the modules exist in the same codebase alongside the core code, but are separated by the import structure. Moving them out would leave just the core parts, and the modules would be separate that you opt into, including into a game should you need them.

core requirements
There is then in fact a list of design requirements and goals for the core itself. This is part of the vision of the engine, where I set out to solve ____ for myself by making luxe at all.

Some of these requirements are defined at a higher level at first without detail, like the idea of a scene, and some of these are defined more specifically, like how the parcels work with assets because I've done the exploring for that in previous engines and games. The unknowns and undefined stuff is why we've been exploring.

Let's break down an example of the typical requirements that the core will need, on a system basis:

• io / assets
• audio
• rendering
• animation
• input
• text
• world

This doesn't include higher level facilitation and helpers like states but the core is constructed around a set of systems that facilitate making games and modules.

The pieces
From this perspective, the current state can be viewed as follows.
Committed below means a closer to final design being committed to code (rather than just design).

Assets have had a good amount of iteration, and are now designed but not fully committed. The audio API has had an iteration to in snow alpha 2.0, so it's quite close to committed. The input has some design exploring going on (Tilman has been exploring input contexts in the repo recently) and we've designed another iteration that isn't committed yet. Text is only partially committed, but its design is fleshed out, but not in.

Two of these aren't even in the current code (animation, world) except in placeholder form (tweening, scene). Plus of course, all the work on embers has been in the design cave, exploring the rendering architecture in code while coming up with something that can be committed too.

The hinges
One thing that should stand out is that a lot of this stuff is hinged on rendering. The way the world might work is affected by how it's rendered. How animation works is also related since it's driving visuals.

The path I've taken to the current state has been along designing the non rendering related systems first, while I design the rendering goals. Given that the rendering is a significant portion of the engine, it left space to do the rest while it's thought through and explored. The last few months have been on finalizing that rendering design and has been going quite well.

So what happens when the design for the renderer was ready? It became time to commit it to code - but how does that happen?

Meandering

In practice, it's not unexpected to lose sight of the endpoint briefly and focus on things that are relevant but not necessarily directly contributing to the goal of making the engine. This is even easier when you have a bunch of enthusiastic people making awesome stuff, and game jams every so often, people releasing actual products to end users across numerous platforms - it adds up.

I don't regret any of the time spent on any of these things - I'm definitely very happy with the outcomes so far, and the amount of benefit to the Haxe ecosystem via snowkit and it's initiatives have been exactly what I had set out to do and continue to become better, but we are still here for an engine, and I want to finish that (so I can make games with it).

distractions
There are also things that I shouldn't have spent time on that I have been, things that take time away from the core can obviously be pretty distracting. Because they're easy it's fun to work on them, and they make people happy.

At any given time though, there are 1000 of these things and none of them move the engine toward being done. That's not even counting necessary distractions like Haxe or hxcpp updates breaking things for users or VS being a mess and so on. It's easy to move around a lot but it can end up seeming like going in circles, and the longer the core isn't done the more that happens.

reset
It was quite important for me to just stop to breathe a bit. The rendering gave me a good pause to do this because I could focus on the one thing, and is the last remaining major system with some unknowns. With the unknowns resolved, it would be time to commit the core design and move forward. Luckily I caught some of these distractions early enough, so I could avoid the typical infinite treadmill any further.

I won't be meandering any more until the engine core is complete,
which means expectedly some things will get less attention or appear slower. This shouldn't concern you but it helps to be clear about what's happening. A common statement I've made is that some things are the way they are as a side effect of timing. The time now, is to finish the core parts.

Ultimately it's the best thing for the engine, even if some things are awkward a little while longer or whatever, I have to make the important choices needed to realize the vision - i.e why we are here in the first place - the engine.

The core ingredients

So a little while back, I'd finished the renderer design and stepped back a bit to assess what it will take to bring it into the current core. Within minutes it was obvious that this wasn't going to work well at all, the current structure is largely predicated on how phoenix worked and embers doesn't match the same model all that much.

I sat down to write out all the requirements for the actual core based on the vision, and then wrote down all the design outcomes of the exploring to see what pieces were missing. What we learned, what wasn't useful and what was. This was really helpful, since it let me view the design goals again, let me clarify my view of the vision and the ultimate outcome, outside of just viewing it as a work in progress code base.

practical example: Luxe.draw
To give an idea of some of these things, let's assess the draw API. The Luxe.draw endpoint is a placeholder that is specifically to debug and draw stuff quickly. So what did we learn from using it in practice, and how would it look in the final version?

Well, I don't enjoy micro managing lists of geometry that I know are disposable. I know that for performance it makes sense to collate all the debug things into one geometry, and not create and destroy them every frame, but this isn't always practical for a number of reasons. This makes it slightly awkward since I usually end up with a mini manager above the drawing to wrangle the geometry.

The second issue is that it also serves as the way to create visuals for sprite-like geometry usage. So in practice, the current API is bad at both of these things, it doesn't work nicely for either. How do we fix this? Well they don't need to (and shouldn't) be coupled, there should be a good way to create sprites based on geometry primitives, AND there should be a nice way to draw debug stuff.

I've got a really neat design in place now for both of those and will expand on them in the dev posts, but it would be nice if there was a way to associate the debug stuff with the world, since that's usually where you want them to be drawn. But, which world right? This brings up the next example design issue:

practical example: luxe.Scene
The requirements for the scene are sort of well defined in design but aren't committed to code, partly because of the rendering hinge, but also because of first solving the balance between the foundation that needs to exist and the being able to do what your game needs without it getting in your way. Or worded differently, over flexibility getting in the way of core design requirements.

What is meant by the foundation is literally something to build upon, something that modules can build on. Something that's consistently the same and reliable.

Say you found some module for the engine that did path finding in the world - in the current system that "world" doesn't really exist, it only half exists. It would be annoying because people would be creating "shared" solutions for their perspective of what the world looks like and it will likely be incompatible with other modules that rely on their own idea of this half-world.

This would mean that the design we commit to has to push the world concept into the foundation first, otherwise it won't facilitate that design requirement well. Doing this solves a myriad of other design issues too, which I'll cover in detail in subsequent dev posts about the solution I ended up going with.

Just add water?

Now that I had a good handle on the road forward, it made sense to try and see what the core would look like, written from scratch. As with snow before I started a blank repo - just to mess around in - and was just throwing the ingredients in and seeing how to ideally fit them in place. This is like sketching where, I don't have any expectations I just need to see it first.

I had a nice list of the systems, their requirements, all their design questions and answers, and I had the vision and design requirements fresh in hand - so I just needed to know how to piece them together in an elegant way.

It's not as simple as just throwing them in and just adding water (it never is)! It requires asking some important, and sometimes difficult questions to get to the answer. A good example is,

What does luxe look like at 1.0?

This is something I could describe easily in requirements and goals, but not in details, until now. The exploring phase has touched on all the systems and I have a good understanding of what each system looks like, I even have a number of them explored out in code in order to be sure (as you will see at the end of the post).

This question is more nuanced though, especially when you ask it standing at the end of the journey looking backwards. If luxe was done today, what would it look like day to day, hour to hour? What does the user do, while making games over months? Or jams? What's left to resolve then in the design, for 1.0.0 to be crystallized? This is the higher level stuff that makes the experience and ties together the systems, it's not just code architecture.

It's where a lot of difficult questions get asked about aligning the technology and the systems with the requirements and vision. There are humans on the other end, and that's pretty important to remember.

ranged sight
This is where a strong vision can be critical, if you don't know what you're trying to do, it will be pretty tough to solidify to anything specific. In our case, we have specifics in mind, and I'll be expanding on this over the next few weeks right here.

For an example, one of the core design requirements is rapid development. This translates to rapid iteration which in practice relates to things like: scripting, data driven content, and hot reloading.

How long does it take to exact a change and see it in the game? At 1.0.0, does the user actually have to compile c++ every. single. time? How much of their game and the engine can be driven by data? How much of the engine can be reloaded when the source inputs change? You should be able to spot a number of these concepts in the current code design already in various ways, waiting for their time to support this requirement.

coalescing
This is how we finalize things, by unifying the code design with the core principles and express that through our malleable tool of code. It's how we get from here to a beta.

There is a reality here though: all of the requirements the engine needs in systems and in design goals are going to require answering the hard questions in practice. How we answer these shapes luxe 1.0.0 and determines whether it's sustainable long term.

Light at the end of the tunnel

Even though it feels a bit slow on the user side at the moment, the last month or two and the rendering design phase before that have been extremely refreshing.

I've been able to reset my view of the vision and goals, been able to identify the things that are keeping me from finishing the engine, and been able to solidify the systems design for the beta. I can see light at the end of the tunnel and I'm relatively certain that the engine is going to be what I had set out to make.

It might not look like what you (or I) expected or assumed, but I know it will be what I wanted to make, and what I need. I like to remind myself that luxe is first a personal game engine that I'm making myself. Something that I need to make the games I want to make, and something I've worked toward for a long time.

I love working on it in the open while I figure out what that looks like, and I love that people are empowered by the work I do - but it would be pretty silly to work hard on something and not see it through.

In simpler terms: I'm not going to compromise on the vision right before the it's done, so I'm going to make the hard choices in stride and do what is needed to achieve that.

Expect significant changes

Since the design has been formalizing it's a bit easier to see how the engine is going to look and feel and behave. It's definitely worth adding a note here that there are heavy changes expected (which shouldn't be TOO surprising).

Obviously changing how the rendering works will have a big impact, since it's underpins a lot of the engine, but bringing all of the core ingredients together into 1.0.0 means discarding everything that doesn't contribute to the vision and goals.

I'll go into details about these things in coming posts, but it's useful to consider that I'm giving some (more) warning. The alpha label was chosen pretty intentionally, but I think this large refactor is going to be breaking in significant ways. I would go as far to say that it might not be backward compatible at all, so please keep that in mind.

:o
Before you panic about that: The existing code is not going to stop working because I've said that. The existing code is not going anywhere. The existing code still works, and if it's solving your needs your needs are still solved by it and it's still usable.

I'll support the existing code for as long as makes sense, especially since Operator Overload is releasing on Steam in the near future, and there are other big games on the current codebase - I'm not going to pull the rug or delete the code or anything crazy.

Being clear about this ahead is just better, even if it deters some users. Going forward for a bit, what you will see in the repo is a bunch of tidying up, fixing obvious stuff and unifying the documentation - but no significant code changes are expected in the existing branch for the immediate term. All the other libraries will continue as mentioned - as those aren't luxe specific.

I'll talk about the new code very soon though, it will all be opened up as usual etc.

Ahead

I'm quite excited to start talking about how the final design looks, and talk about some of the answers to those design questions. But I'm also quite excited to start using it more for games.

Since I've been exploring in the design cave, I do have things that I've answered and committed to code, with the help of some core contributors (Tilman has been helping a lot!) so we've been making significant progress.

I've got a bunch of posts talking about this lined up, but here are some teasers on what you can expect soon:

Lines with cap, bevel, thickness, and feathering for smooth AA.

Text with multiple fonts, colors, sizes in a single draw, using msdf.
These fonts are loaded from a ttf file in the project.

Animation driven by curves (the gif is lagging not the animation).

fin

This above stuff is just the design explored in practice, and it doesn't even show much. The way the parcels are handled, the way rendering works, the way the systems tie together, the fact that some above examples are specified by data files, the way the worlds and levels work, where templates and sane defaults fit in - there's a lot to talk about!

Taking an entire project worth of design to committed code is a rapid and rewarding process seeing everything come together - I think if you're here for the engine, you will be just as excited with the outcomes too.

I'm going to go into full details about what the design cave has yielded,
and I hope to see you at the next dev log where I'll start breaking down the systems and what their requirements and design has become!

Feedback on this post is below or can be found in this discussion.