Today we get to talk about a topic everybody loves to get excited about - programming languages! More specifically, we're going to talk about the needs when making games with luxe, the landscape of language choices, and the goals of the project being the most relevant decider.
Note that the 1.0 version of luxe discussed below is not available yet,
we're working on getting it usable! Subscribe for more news.
retweet this!
operator overload
Benn's game operator overload was alongside Westport Independent as one of the very early few who put their trust in me and luxe, and worked on a big project early on. It's the last of the bigger projects shipping on the alpha codebase, which is good too since we're moving forward!
And now it makes me really happy to see this game shipped! It's been released now and is on Steam, and if you like very well designed puzzle games you should definitely check it out and tell your friends. Retweet this if you can!
Not coincidentally but maybe surprisingly, I worked on the soundtrack for the game, you can read about it here.
Right so where were we... oh yes, onward!
languages are tools
In practice, a programming language is a tool. It's a way to express an idea in a concrete way, typically to solve some problem or need. We all know that the best tool for any task, is the best tool for that task...
What that means in practice is context specific - there is no one definitive tool for every task obviously - so there is a pragmatic decision made in order to solve for your requirements.
For interest sake, on an average random week I use several languages - c++, c#, javascript, haxe, lua, python - plus other "smaller" languages like hlsl or glsl. That doesn't include visual programming tools like Unreal Blueprints, smaller DSL's (domain specific languages), or even my own languages that I've made. It also doesn't include spare time tinkering, where I'll mess with other languages like swift, go, rust, nim and any of the hundreds of home made languages on github. All this alongside whatever else I feel like playing with at the time.
This shouldn't surprise most people, many programmers I know treat a language as a means to an end. Programming isn't defined by the language that it happens in, it's much more than that. In reality a programming language is simply a tool, and the tool is use case specific. The tool that fits my needs, and which tools are made available for that need at the time.
I'd say my weekly list may be a bit longer than average, maybe because I've been programming for 18 years and have an interest in languages, or because I work on cross platform projects, but many developers do similar without realizing how many languages they interact with.
languages are hard
Creating and choosing languages is about compromise. It's also about accepting the confines of reality at the time, the language you consider absolutely perfect probably doesn't exist. (And your definition of perfect probably doesn't align with mine).
Many programmers unfortunately aren't realistic, or miss a lot of important context when discussing programming languages. It can lead to a lot of passionate but misguided discussion, typically involving someone championing one language as the only usable one. We all know that one tool doesn't fit every need, so this is inherently silly, but most importantly we're just not at the point in history where a lot of the things we care about are solved.
They're also not simple to solve. Languages are hard, and there are countless amazing engineers dedicating their lives to explore this stuff. I am not one of them.
I'd like to quote something Graydon Hoare (creator of rust, now working on swift) said in a recent post about the state of modern languages and what's next for them, emphasis mine:
(Writing this makes me think it deserves a footnote / warning: if while reading these remarks, you feel that modules -- or anything else I'm going to mention here -- are a "simple thing" that's easy to get right, with obvious right answers, I'm going to suggest you're likely suffering some mixture of Stockholm syndrome induced by your current favourite language, Engineer syndrome, and/or Dunning–Kruger effect. Literally thousands of extremely skilled people have spent their lives banging their heads against these problems, and every shipping system has Serious Issues they simply don't deal with right.)
In practice, all languages have to face this reality. As a programmer, so do you. The way I tackle this is by being pragmatic about my requirements, and finding (or making) the best language for that need.
This applies to luxe too!
unrelated: some node graphs I've been playing with.
pragmatic language requirements in luxe
So we have a project that needs a default user facing language...
Typically we have a list of pragmatic requirements that need to be satisfied. For example, as discussed in the last post - C/C++ is the most portable language for the core runtime and satisfies several constraints making it the best tool for that task. That's for the core - not the average user. The average user doesn't want to program in C++, the user facing language has different requirements.
Since we're busy implementing the fleshed out design of the long term version of luxe, as with all aspects of the engine, we need to consider the design goals and align them with the user facing language.
This is the language 99% of users will be using day to day to make their games - What do these requirements look like? An incomplete list and in no particular order:
- lightweight
- fast iteration time
- high level and modern
- write-once portability of game code
- consistent - no breaking while I am asleep
- building - no platform specific compilation steps
- fluidity - quick expression of ideas/intent is key
- design - align with luxe design philosophy - simple/elegant
- familiarity - obscure languages aren't ideal
- flexible - high engine <-> language cohesion is great
- reasonable performance - good enough for 99% of game code
- easy interoperability - with lower level core and more
There are more requirements, like idealistic ones and preferential ones , however the ones above are closer to fundamental requirements that align with the luxe design philosophy that are ideally not debatable. They directly affect the user workflow, something luxe cares a lot about.
seemingly-related-but-not-really requirements
Since luxe is cross platform, the language has to be portable. Since luxe is a game engine, there is sometimes a need for higher performance code. It may be tempting to try to solve this using the same solution but that could easily compromise many of the requirements - for example will 100% of users have a need for performance sensitive game specific code every day? No. In the luxe design philosophy we don't believe in 100% solutions in general, so this wouldn't be an ideal approach.
Instead, there are two problems here, with different requirements. That means there needs to be multiple tools to address our needs. As an example, if the engine provides the ability to use native plugins in a DLL via a C api - that would be a tool to use for performance sensitive code, and wouldn't adversely affect the other requirements of the engine.
These types of tools compliment each other. They work together to solve our goals, giving us ways to express our intent and ideas - which is what luxe is about.
A clip from Tilman's game project using luxe 1.0 stuff
pragmatic life choices
I make games. I make luxe as a way to make games quicker, with my own workflow. I am not a dedicated language engineer spending my life solving problems decades of engineers haven't yet. I am not going to be tackling any major language design problems any time soon. This makes it easy to make pragmatic choices about how I spend my time. I don't have to worry about it!
What I do have to care about is that I'm able to make the games I want to, whether the workflow suits my goals and preferences, and let's me do what I need to do. I must worry about the day to day, and the foreseeable future for luxe. I'll be solving that equation by picking a language that fits the workflow requirements, and that's a lot easier when you know exactly what you need - like we do!
It also means I don't have to debate the merits of my choices with a million other programmers who have their own needs, goals and perspectives :)
Technical weight
There's a lot more than just the requirements that go into the choice of a language, such as the consideration of technical weight - a great post
from Bart Wronski. As a (mostly) solo developer on a game engine, I have to make choices very carefully. Every single one of them fall on me, and some options are like going hiking with a small town strapped to my back, something I have no interest in doing.
great defaults
Phew, so yes we have a need for a default language, the canonical luxe experience for the 99% of us that just care about making games. As I said in dev log #3 and #4, it's a high level scripting language that fits as many of the ideal requirements as possible.
This language will also serve aspects of the engine level workflow that benefit from being flexible. Like project stages (pre and post hooks), custom user content types for the asset pipeline, scripted rendering control flow, there are many fundamental parts of the engine workflow that need a nice fluid language with the same requirements.
What does that language look like?
hello Wren <3
I've been looking for a language like wren for a long time, so imagine my delight when I stumbled on it while reading the author's blog about language programming.
Not only does it align with every requirement, it also aligns with a lot of my personal preference and my idealistic requirements. The lead dev aligns well with my values (and wrote a book on game programming patterns), and from the moment I first saw it years back I knew I had to try it out for luxe.
When I started the final implementation of luxe, it started with wren at the heart of the workflow. and has proven to be a great choice. I've written at least 50k lines using it and I have no regrets. I love using it daily, and have for over a year.
The whole package
From the way the module system works, the design of the language, the size of the code, the fibers, the amount of control the engine can have over a cohesive experience, the surface area of the API, the fact it's designed well for static analysis, the list of things I like about it are fairly endless. It would take a long time to go over all of those things - I'll go over more of them in time.
So it's perfect?
No! No language is obviously, but wren is the closest I've found that matches what I would write if I made a language myself - except it's made by a language engineer with over a decade of experience. The 99% of stuff that is there is absolutely great, and for the other 1%, well we have the means to do what we need to with it.
Bob is currently busy writing a book on crafting interpreters, so his attention for wren is lower right now. This doesn't affect us much because the language has been largely stable, and that type of consistency is exactly what luxe and I prefer.
However I've been talking to Bob for a while about wren and have been contributing what I can, so I have no concerns about that. This is one of the reasons wren fits well too, if Bob isn't around, or in the extreme case of Bob moving on, we can just keep improving and growing the language ourselves - all the important work has already been done.
We already have some small luxe specific workflow tweaks in there but there are some things that benefit all wren users that we've also been working on.
Trajectory
In dev log 4 we spoke about things that are good now, but get better over time. Wren is one of those things, where it's very close to what we want but it still has a wishlist of things we'd like, things I'd consider ideal to have.
What are some ideals that wren doesn't have yet, and what can we do about it?
Ideal++: static checking
I love dynamic languages for high level coding (there's a reason they're prominent in scripting!) but I also like compile time checking for obvious mistakes. This is an one example of why I love wren, due to the way wren is designed, it's easy to have both dynamic fluidity and compile time checking.
We already have some of that in place, whenever you run your game errors will be pointed out before hand. There's more to go on from there...
Because we have access to the AST from here, we can do all kinds of compiler stuff like code completion, reference locating and so on later. Let's say in future we have some form of type annotation, we can easily do type checking (and other forms of static analysis). Some choices are already great now, but get even better as time goes on. Many decisions made now are for the long run!
Ideal++: debugging
I know many people prefer working in a language with a debugger, so while wren doesn't have a debugger in master just yet, Tilman (KeyMaster) has been adding one to the luxe branch.
We'll have great debugging support in luxe before it's in your hands.
Here's a preview of it working inside vscode.
In that video you can also see the shape casting collision stuff Tilman has added as well!
The implementation of wren affords that someone with no real experience in the wren runtime can implement full networked debugging support in a few days of effort, and hook it up to vscode. This is a powerful relationship to have with the language in the engine, and another reason why I love it.
The age old catch 22
With projects like wren and luxe, there's a (valid) tendency for people to want to see other people using it, as a sort of social proof, before they'll consider it.
This comes up a lot but it's obviously a trap, we'll use the engine if others are using it, but someone has to take that first step. This isn't unique to projects like this - “We'll give you a job if you have work experience, but we won't give you work experience.” I've always found this reasoning to be frustrating, so I've always taken the chance to negate that when I had the opportunity.
With luxe, I had encountered that several times, and for the alpha cycle decided to tackle it head on by getting projects and developers on board early. It worked out well, we now have several games released in the wild on Steam, mobile and web, even on the alpha codebase.
I know some people may feel that taking on a language like wren early is a risk (and there is definitely some amount of risk), but I feel wren deserves something like luxe taking the chance on it. Not as some heroic act, but it's a valid opportunity to negate this type of catch 22. It's specifically because wren fits our pragmatic requirements, almost perfectly capturing what I want for the luxe workflow, and it's well past a lot of the risky aspects that come with the territory due to the author being a language developer.
I already took the chance on wren last year, and have not been let down. I'm looking forward to contributing more to it's improvement and growth via luxe and our community!
what about other languages?
Sometimes I like to use other languages, when my requirements aren't that strict. For instance if I've got a game that I know for sure will only target desktop, it might be fun to write it in rust or c# or something else. There may also be requirements (like porting) where using existing codebases would be good. What do we do in this case?
This is where the c++ core with C API comes in, and where plugins shine.
It will take time for this to be available, but it will be possible to use any language you want with luxe via script system plugins. The API can be bound to other languages easily, and the workflow for that language can be idiomatic. If the language can be used via C API binding you're good to go. As a proof of concept, we've already played around with python, rust and haxe externs.
Here's a post where I wrote about binding a whole lot of languages
In practice however it is unreasonable to imagine I'll be maintaining every language ever, and I care more about the canonical luxe experience being fantastic - so this won't be a focus of the engine but of the community. The engine provides the tools, it's a facilitator after all. (I also prefer that the ecosystem not be diluted too early as this helps nobody).
Wren is the canonical language of luxe
And will serve the workflow and core runtime needs. The 99% of us that care about making games in a fluid workflow will be using that, and down the line people who really want to use their favourite language (or need to use it) have the tools at their disposal. Hooray!
Going forward...
The best part about this post is it removes a lot of the unknowns about the 1.0 design of luxe, and the "secrecy" which was necessary to validate the design choices of the engine. For example, the path we took to get here, based on the exploration of the design before the actual implementation.
click to enlarge
This also means that I'll be able to show WAY more of what we've been working on, explain the design and architecture of the engine in more detail, and put up a lot more demos. Here's one now!
- Requires latest chrome/firefox (runs on WebGL2/WebAssembly)
- This is not a benchmark (but you can see wren in action here)
- [Click here to run the demo](https://underscorediscovery.ca/lxs/bounce/" target="_blank)
There's a lot to talk about!
Now that we have all the puzzle pieces in place, there's plenty to post about that I'll be going over in the next dev logs. Since we're no longer waiting on unknowns they can come more frequently and be more detailed.
We're getting pretty close to a user facing version and I'm pleased with where we are headed.
If you'd like to be notified when these posts are made, sign up here. I'll only send an email when a notable dev log is posted, that's it.
You can follow news about luxe on twitter @luxeengine or by subscribing to the feed or using the mailing list sign up above. You can find that information on luxeengine.com/news
Feedback and questions on this post can be found below or shared in this discussion.