Game rush

TL;DR: I'm starting on a simple html5 game project.

I've decided to make a series of mini-games aimed at html5 compliant browsers. Each game will have a personal goal in familiarising myself with some key html5 technology or support library (like three.js). 

Ok, so here goes. The very first game concept "art"  is for a very derivative game called "Invaders from space! OMG!". It's a working title...

You control the TOSPOT (Tank Of Space Protection from Otherworldly Threats, or whatever other rude sounding anachronism I can think of before I make the game), your planet's last line of defence against the unknown dangers from outer space. The antagonists, being beings from outer space, are clearly more technologically advanced than you and their nimble frames can effortlessly evade your every attack. On their slow, meandering (yet evasive) path down to the planet surface, they shout admonitions and insult your inferior technology. The twist is that the invaders from space, although rude, are peaceful and just want to give you a group hug.

The end.

The personal aim of this project is to familiarise myself with the web audio API defined for html5 and to brush up on the html5 canvas element. This game will be 2d and I'm setting a deadline of 2 weeks for implementation. Let the challenge begin, tomorrow! (I mean, it's Sunday... you seriously expect me to start working on this on a Sunday?)

Change of plans... again.

TL;DR: I'm giving up trying to make an engine for now since better free tools exist already, and they're in better state than I expected. Go look at these pretty demos from other people, or go listen to these demos made by other people, if your browser supports it.

Maybe you spend all your time working on something everyone else seems to be doing, but no one quite got it right, or maybe there exists a solution, but it is out of your price range... Sometimes you could be wasting your time doing something someone else (meaning better connected groups of individuals and corporations) already did, quite a while ago, but you just didn't delve deep enough into the research to find it. And by "you" I mean me, cos that's what I kinda did. Oops. Also, I'm running out of time, and I want to be known for games, and not engines.

I was working hard on creating modular wrappers for audio sources, and most recently managed to add a spacializer class for attaching audio sources to rigid bodies. Being single minded about making a game engine I never bothered to read the W3 specification for HTML5's audioContext object which includes everything I was trying to achieve in terms of audio routing in my own project [1]. Searching for examples of implementations using Chrome as browser convinced me that I was wasting my time [2] [3] [4]. The web audio API is mature enough in Chrome to do everything I set out to do, so I might as well use it and hope that future adopters of HTML5 will provide a half decent implementation of the autioContext specification.

Further down the line, I expect that the audioContext will make its way into QtWebKit. When this happens I will gladly base the next iteration of the engine on QtWebKit to create a self-contained executable.

In my defence, time has moved on since I started the project early last year... The state of HTML5 was matured, but not complete. It appeared to have a bright future, but that future wasn't "now" back then. I don't think we're at the "now" stage yet, but it does seem close with M$'s adoption of HTML5 as an alternative application framework in windows 8.

I had a plan to develop a game engine with emphasis on a robust scripting environment. At the time I was trying to get away from web dev for just a bit. I decided on Qt for its cross-platform Win/ Linux and MacOs support. Somewhere after the Bullet Physics integration I started to address the issue of GUI. I followed the over-painting example from QT to use QPainter for any GUI text. Even later, after doing the OpenGL script interface I wondered if I could not save a lot of effort to just use QtWebKit for the GUI, since I was already well versed in CSS and HTML... it seemed natural. Round about this point I came back to see the state of HTML5 and WebGL, and I decided that for now, they are good enough for my purposes, limited by time as I am. Seems I went full circle to land back at web-dev.

Sigh.

Anyhoo, the video below shows the point my engine got to before I decided to give up making engines.. It demos the audio spacializer somewhat (STK sources get attached to rigid bodies and the mixer parameters get updated from the motion-state callback). I was about to add the delta-Position variable to the bodies' motion-state for Doppler effects, so that is not done. (Also no texture support... the colours are generated by a simple shader):

I plan to open the source for this project soon, I just want to first remove the direct inclusion of STK in the source and link to the library instead. There is some documentation, but it is mostly undocumented. All in all, though mostly a waste of time, it was a great learning experience. Don't hold your breath for the source though... priorities.

I will keep you updated on the progress of my HTML5/WebGL game development adventure.

Links:
[1] http://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#Spatialization-section
[2] http://www.html5rocks.com/en/tutorials/webaudio/intro/
[3] http://www.chromeexperiments.com/webgl
[4] http://chromium.googlecode.com/svn/trunk/samples/audio/index.html

Almost done with audio

TL;DR: Still no pretty pictures. Sorry.

Since last post:

• Mixer.  Per-source volume and pan.  It can have multiple modules as sources, and forces any mono sources to fill all output channels (dependent on pan setting). 
• Stk Voicer support. Can essentially make polyphonic versions of any Instrmnt and trigger it from the script environment.
• Started on Stk Filters, but apart form delay, most of them seem ill-fitted for my purposes (see below).

Currently only stereo linear-cross-fade panning is supported by the mixer module. I'll re-visit multi-channel placement later. Audio positioning for game objects will only rely on volume and pan for now.

What's still left to do?

• Ignore all Stk Generators for now... they are too low-level to merit script instantiation.. They might make a comeback as LFO-type control inputs in the future...
• Ignore most Stk filters. Filters in Stk seem to be meant as helper classes for internal workings in custom Instrmnt derived classes. Many filters do not normalize and without extreme tweaking aren't useful in an intuitive real-time modular fashion. Compromise: Script instantiated filter which can be set up as a HP,BP,LP or BS filter. Internally an IIR Stk filter can be used with a known calculated set of normalized coefficients for Butterworth filters. This will be sufficient for games.
• Ignore score files for now... The effort which will have to go into creating a scoring environment is beyond the scope of a gaming project. It is a love of mine, hence the long interlope. I will come back to this, but not yet. 
• Ogg file loading/playback for music needs to happen (Easier to compose music in Renoise for now).
• Stk Effects to support : Chorus, JC-Rev, Echo ... the rest can come later.
• Implement tying motion-states to audio sources.

Tick these off and I'll have enough audio tech to my liking. 

After that? Menu GUI and exposing the keyboard-map to the script environment, then we'll be close to ThingMaker version zero point something!

Ok, so it is still a way to go, but an end is in sight.

No pretty pictures here, please move along...

TL;DR: I made a sound from within the scripting environment, and it was hard!

Disclaimer: programmers only.

It has been almost a month since the last post and no "visible" progress has been made...
Lots of code have been committed for the purpose of making a couple of extra lines of script work.

Basically, I had to write an interface class for an abstract "sound module"... which is something with an internal audio buffer and references to inputs and outputs to other sound modules. Script interfaces had to be written for connecting modules together with logic to detect any feedback between modules. The only sub-classed objects so far is a master module, which abstracts the "default" audio output device, and a Stk Instrmnt wrapper class.

When I designed for using Stk, I planned around the Stk base interface classes as much as possible, and being obsessed with the sound module idea I settled for the most common audio processing virtual function shared by all higher level Stk interface classes... namely : StkFrames &tick(StkFrames&frames).

Since this most common tick function processes a block of audio frames, I thought it best to have the algorithm which processes the audio frames in as large a chunk as "real-time" permits... Stk defaults to 512 frames, which at 44k1Hz equates to about 11ms (good enough for games!).

Wasting of time, feeling stupid and doing more work than necessary happened when I decided to design for feedback support after already having "chunk" optimizations in place.

What am I talking about? Say you have an instrument which you plug into a mixer. That mixer goes through an EQ and into a splitter. That splitter then has an output to a chorus effect and another to a re-verb effect. The chorus goes to the master output, but the re-verb goes back to the mixer the instrument is plugged into. Why do you want to do this? Well, why not? It might sound cool... who knows, but the point is you might just NEED it somewhere.

The problem is now apparent since it is clear that the mixer's second input is dependent on the output of some later stage which is in turn dependent on the mixer's output. The best you can hope for in a digital system is to allow the mixer to do a single frame's tick, propagate that tick though all modules until you update the module the mixer input is dependent upon which will then serve as the input to the mixer's next single-frame tick operation.

So, if like me, you went ahead optimizing for chunks, and then wanted to go and have a special feed-back case which just does single ticks, then you're probably also half-wasting your time (I'm sure it's possible, just probably not with the way I went about it.).

What I kept from this misadventure was just the ability to detect if there is feedback when a connection is created, and throw a script exception if so, not allowing the connection. A lot can be done still with just feed-forward modular audio networks, so apart from the wasted time, I'm not too bumbed.

So, I now have the master module which is the entry point for connection to other modules and the Stk Instrmnt wrapper  (which currently supports all the sub-classed instruments Stk has to offer). Further sub-classed modules still need to be added to complete the whole sound experience, but the minimals are coming together nicely.

The basic functioning of the audio engine is like:

• audio callback requests buffer
• audio engine processes the master module
• master module processes its source modules recursively

...still not noticing any non-stupidity-induced breaking sound artefacts, so all is still well and real-time(ish).

Hopefully the next update will have some video with "actual" sound. In the distant future, I hope to re-visit the modular feedback issue. 

What went wrong this time? (Adventures with STK)

Sorry, no pretty pictures in this post... Also, this post is moderately technical in nature, so browse away now if you are not interested in audio-programming jargon. You'll be happier for it. If you know Qt and STK well, you might find this post slightly amusing. If you are new to Qt and STK, you might not know what I'm talking about at all. This post is personal, I guess.

I've made the first steps for physics, interaction and graphics, and since I'm interested in making desktop games, it is only natural to do audio at some point...

I thought it would be easy to integrate Qt's audio functionality when using the qmake tool-chain... add qtmultimedia to the config in your .pro file, and include a QAudioOutput class somewhere and trigger sounds from that... Right?

Well, it kinda is... but not really... QAudioOutput is a reasonably low-level interface class and can read from a custom QIODevice derived class (pull mode, good for streaming... limited real-time application) or you are provided with a QIODevice which you can write to once a notify interval has elapsed (push mode... more real-time).  A QIODevice "talks" in raw bytes... so either way you still need to take care of formatting your audio data correctly to match the stream parameters (code for this can be seen in the Generator::generateData method in the Audio Output Qt example ). Also, no mixing/routing functionality is provided... you're expected to implement that or use some third party library like STK. So, although the QAudioOutput class provides a solid cross-platform starting-point, it still requires you to implement a lot of the low-level, broiler-plate, byte-array, endian-conscious code. Also, this part of the Qt framework is under active development so it might drastically change  in the main Qt distribution as projects become merged into it. It is inevitable that technology will pass you by... but it is all part of the learning process.

For a while I've wanted to use Perry Cook & Garry Scavone's STK (https://ccrma.stanford.edu/software/stk/) to provide the basis for a soft-synth audio framework for my game engine. As a preference I decided on RtAudio and STK over PortAudio and CSound, though I know much is interchangeable and CSound has a great deal more features. I was drawn in bySTK's well balanced compromise between latency (#1 consideration for games), footprint and readability.

The aim was to have an audio engine class which gets instantiated along with the script environment, which would also provide script interfaces into a range of STK classes and to dynamically connect them in a modular fashion... Since STK can be compiled without RtAudio dependence I though it best to stick with the QtMultimedia module since I was relying heavily on Qt already.

One can conceivably tie STK's non-real-time build in with a push mode implementation involving QtAudioOutput, but, after wasting much time, in the end experimentation with RtAudio  made me decide to use it instead of QtAudioOutput. The biggest factor for me was that RtAudio already deals with floating point samples and transparently does the conversion to raw audio for a wide range of audio formats. Also, it has a relaxed open source license and is cross-platform (Linux/Mac/Win). My DSP courses dealt mostly with audio signals represented as numbers in the range -1.0 to 1.0, so RtAudio is a more intuitive fit in this regard also.

On the plus side, choosing RtAudio over QtMultimedia also provided the additional ASIO support in windows ( although ASIO support is compiled and tested, I'm  defaulting to dSound on Win32 platforms for now) and RtMidi cross platform MIDI IO support ( which has also been compiled in but is waiting untapped). The STK source files have been statically incorporated into my project not because I planed it, but because I was initially only using some elements of STK... Eventually all of STK source got included. I might eventually move back to just linking to the STK library. Right now I don't care much because the code footprint is small and all the STK classes are pre-compilable, so they don't affect consequent compile times much (exception is lesson 1).

Lessons learned:

1. Complacency can make you forget the classics: Changed a define in your project file? CLEAN! REBUILD! (And don't forget the cleaning part. Too embarrassed to say how much time I wasted dealing with puzzling bugs after forgetting to do this).
2. RtAudio's closeStream() method should never be called from a destructor of a class your RtAudio instance is part of... safer to have a pointer to a RtAudio instance which you manually delete after step 3.
3. You need the "bool done" tick-data element you see in all the STK code examples... no escaping it if using the tick() callback functionality. In the main tick() callback check for data.done and return -1 if it is true; The audio engine class destructor should set the "data.done" value to true and should sleep for a couple of microseconds to allow any pending callbacks to return a -1... only then is it safe to delete the RtAudio instance without a hang (on my system at least).
4. Building STK with Qt(Qt Creator/qmake) MINGW: If like me, you added all the stk/include and stk/src files to your Qt project, you'll need to add some defines and library dependencies to your .pro file to get everything linking and compiling right. To get the compiler setting and project defines for STK, run STK's configure script from within a msys shell to generate a makefile. Reading this makefile will tell you what defines and libraries you need to add to your Qt .pro file (you can also look at the VC project files that come with STK but it is more verbose and the linker syntax is different). 

So, what do I have?

Not much really, just getting everything compiled together and first steps of simple integration for now. Tones are generating; instruments are playing; filters are filtering; effects are effecting even... but only C++ tests hacked into the audio engine's initAudio() method for the interim. The initial tests show promise. No audio script interfaces or script definable modular flow graphs have been implemented yet... that is next.

I still have a lot to do with the sound-programming before my next update... sorry, I know I can ramble sometimes. This post went on for longer than intended.

Useful links:

• http://www.mingw.org/ (msys doesn't come with Qt Creator installer)
• https://ccrma.stanford.edu/software/stk/ (STK Home page)
• http://www.music.mcgill.ca/~gary/rtaudio/ (RtAudio Home)
• http://web.mit.edu/carrien/Public/speechlab/marc_code/ADAPT_VC/rtaudio/doc/html/ (MIT RtAudio Tutorial)
• http://www.music.mcgill.ca/~gary/rtmidi/ (RtMidi Home)

What have I been doing so far:

The video shows the summary of my progress on my ECMA-based 3d engine called "ThingMaker" (for now).

It is not impressive to look at for some reasons. You're seeing a early-stage, work-in-progress debug-render of a series of tests, so there is no attempt at aesthetic appeal... The screen recorder program slowed the frame-rate down considerably... I had to compress it a lot cos my up-speed is very slow... This site's video up-loader compressed it even more... So, that's why it is so ugly.

The video starts with the engine initialized. Next I run the test script. Everything in the video is dynamically created via the script environment. There are some script-defined geometries(floating shapes), some physics objects drawn in lines(the pool table), and physics shapes with dynamically created geometries(the reddish shapes with lines around them).

The tools used to create what you see: Qt framework, Bullet Physics.

Behind the scenes:

• Script binding of basics of the Bullet API (no soft-body or constraint support yet, and only basic convex collision shapes supported).
• Script binding of OpenGL 3.3 geometries and shaders.
• Script definable collision callbacks.
• C++ interface for physical controllers, or they can be defined in script. 

...but it is still in its beginning stages. I still require rudimentary audio support in the script environment and some simple GUI elements before I can make a rudimentary game (yeah, rudimentary games with OpenGL 3.3 support, I know it sounds like a joke, but it is part of a "plan"). I left these for last since I know Qt has enough support functions to make this easier than integrating Bullet and getting OpenGL 3.3 core profile to work properly... I hope.

Much work is still needed on expanding Bullet API support and replacing the rudimentary render list with a proper scene-graph... also, only debug builds are currently maintained and the deployment strategy is somewhat flawed. I'll save that for another post entitled : "the ways in which I waste effort"...

Welcome back.

Long ago I was part of a game development house, working as the tools programmer. The work was satisfying, but I had to leave due to financial constraints. Over the past decade I have made many attempts to make games while working "real" jobs. There was always some reason why I couldn't finish making a game - time constraints, job stress, me being an idiot, lost hard drives, technology passing me by...

This February I made another attempt. It has been 9 months of in-between-projects work and I've finally typed up enough code to resemble the beginnings of a script-capable 3D game engine.

Still no games though... but they are coming... oh yes.

The intention of this blog is to share any progress I'm making and to discuss some of the thoughts behind some design issues I might need to get out in order for me to understand myself better.

Mostly though, I hope that sharing my progress will serve as motivation for myself to get the project done and start making some games!