Almost done with audio

TL;DR: Still no pretty pictures. Sorry.

Since last post:

• Mixer.  Per-source volume and pan.  It can have multiple modules as sources, and forces any mono sources to fill all output channels (dependent on pan setting). 
• Stk Voicer support. Can essentially make polyphonic versions of any Instrmnt and trigger it from the script environment.
• Started on Stk Filters, but apart form delay, most of them seem ill-fitted for my purposes (see below).

Currently only stereo linear-cross-fade panning is supported by the mixer module. I'll re-visit multi-channel placement later. Audio positioning for game objects will only rely on volume and pan for now.

What's still left to do?

• Ignore all Stk Generators for now... they are too low-level to merit script instantiation.. They might make a comeback as LFO-type control inputs in the future...
• Ignore most Stk filters. Filters in Stk seem to be meant as helper classes for internal workings in custom Instrmnt derived classes. Many filters do not normalize and without extreme tweaking aren't useful in an intuitive real-time modular fashion. Compromise: Script instantiated filter which can be set up as a HP,BP,LP or BS filter. Internally an IIR Stk filter can be used with a known calculated set of normalized coefficients for Butterworth filters. This will be sufficient for games.
• Ignore score files for now... The effort which will have to go into creating a scoring environment is beyond the scope of a gaming project. It is a love of mine, hence the long interlope. I will come back to this, but not yet. 
• Ogg file loading/playback for music needs to happen (Easier to compose music in Renoise for now).
• Stk Effects to support : Chorus, JC-Rev, Echo ... the rest can come later.
• Implement tying motion-states to audio sources.

Tick these off and I'll have enough audio tech to my liking. 

After that? Menu GUI and exposing the keyboard-map to the script environment, then we'll be close to ThingMaker version zero point something!

Ok, so it is still a way to go, but an end is in sight.

No pretty pictures here, please move along...

TL;DR: I made a sound from within the scripting environment, and it was hard!

Disclaimer: programmers only.

It has been almost a month since the last post and no "visible" progress has been made...
Lots of code have been committed for the purpose of making a couple of extra lines of script work.

Basically, I had to write an interface class for an abstract "sound module"... which is something with an internal audio buffer and references to inputs and outputs to other sound modules. Script interfaces had to be written for connecting modules together with logic to detect any feedback between modules. The only sub-classed objects so far is a master module, which abstracts the "default" audio output device, and a Stk Instrmnt wrapper class.

When I designed for using Stk, I planned around the Stk base interface classes as much as possible, and being obsessed with the sound module idea I settled for the most common audio processing virtual function shared by all higher level Stk interface classes... namely : StkFrames &tick(StkFrames&frames).

Since this most common tick function processes a block of audio frames, I thought it best to have the algorithm which processes the audio frames in as large a chunk as "real-time" permits... Stk defaults to 512 frames, which at 44k1Hz equates to about 11ms (good enough for games!).

Wasting of time, feeling stupid and doing more work than necessary happened when I decided to design for feedback support after already having "chunk" optimizations in place.

What am I talking about? Say you have an instrument which you plug into a mixer. That mixer goes through an EQ and into a splitter. That splitter then has an output to a chorus effect and another to a re-verb effect. The chorus goes to the master output, but the re-verb goes back to the mixer the instrument is plugged into. Why do you want to do this? Well, why not? It might sound cool... who knows, but the point is you might just NEED it somewhere.

The problem is now apparent since it is clear that the mixer's second input is dependent on the output of some later stage which is in turn dependent on the mixer's output. The best you can hope for in a digital system is to allow the mixer to do a single frame's tick, propagate that tick though all modules until you update the module the mixer input is dependent upon which will then serve as the input to the mixer's next single-frame tick operation.

So, if like me, you went ahead optimizing for chunks, and then wanted to go and have a special feed-back case which just does single ticks, then you're probably also half-wasting your time (I'm sure it's possible, just probably not with the way I went about it.).

What I kept from this misadventure was just the ability to detect if there is feedback when a connection is created, and throw a script exception if so, not allowing the connection. A lot can be done still with just feed-forward modular audio networks, so apart from the wasted time, I'm not too bumbed.

So, I now have the master module which is the entry point for connection to other modules and the Stk Instrmnt wrapper  (which currently supports all the sub-classed instruments Stk has to offer). Further sub-classed modules still need to be added to complete the whole sound experience, but the minimals are coming together nicely.

The basic functioning of the audio engine is like:

• audio callback requests buffer
• audio engine processes the master module
• master module processes its source modules recursively

...still not noticing any non-stupidity-induced breaking sound artefacts, so all is still well and real-time(ish).

Hopefully the next update will have some video with "actual" sound. In the distant future, I hope to re-visit the modular feedback issue. 

What went wrong this time? (Adventures with STK)

Sorry, no pretty pictures in this post... Also, this post is moderately technical in nature, so browse away now if you are not interested in audio-programming jargon. You'll be happier for it. If you know Qt and STK well, you might find this post slightly amusing. If you are new to Qt and STK, you might not know what I'm talking about at all. This post is personal, I guess.

I've made the first steps for physics, interaction and graphics, and since I'm interested in making desktop games, it is only natural to do audio at some point...

I thought it would be easy to integrate Qt's audio functionality when using the qmake tool-chain... add qtmultimedia to the config in your .pro file, and include a QAudioOutput class somewhere and trigger sounds from that... Right?

Well, it kinda is... but not really... QAudioOutput is a reasonably low-level interface class and can read from a custom QIODevice derived class (pull mode, good for streaming... limited real-time application) or you are provided with a QIODevice which you can write to once a notify interval has elapsed (push mode... more real-time).  A QIODevice "talks" in raw bytes... so either way you still need to take care of formatting your audio data correctly to match the stream parameters (code for this can be seen in the Generator::generateData method in the Audio Output Qt example ). Also, no mixing/routing functionality is provided... you're expected to implement that or use some third party library like STK. So, although the QAudioOutput class provides a solid cross-platform starting-point, it still requires you to implement a lot of the low-level, broiler-plate, byte-array, endian-conscious code. Also, this part of the Qt framework is under active development so it might drastically change  in the main Qt distribution as projects become merged into it. It is inevitable that technology will pass you by... but it is all part of the learning process.

For a while I've wanted to use Perry Cook & Garry Scavone's STK (https://ccrma.stanford.edu/software/stk/) to provide the basis for a soft-synth audio framework for my game engine. As a preference I decided on RtAudio and STK over PortAudio and CSound, though I know much is interchangeable and CSound has a great deal more features. I was drawn in bySTK's well balanced compromise between latency (#1 consideration for games), footprint and readability.

The aim was to have an audio engine class which gets instantiated along with the script environment, which would also provide script interfaces into a range of STK classes and to dynamically connect them in a modular fashion... Since STK can be compiled without RtAudio dependence I though it best to stick with the QtMultimedia module since I was relying heavily on Qt already.

One can conceivably tie STK's non-real-time build in with a push mode implementation involving QtAudioOutput, but, after wasting much time, in the end experimentation with RtAudio  made me decide to use it instead of QtAudioOutput. The biggest factor for me was that RtAudio already deals with floating point samples and transparently does the conversion to raw audio for a wide range of audio formats. Also, it has a relaxed open source license and is cross-platform (Linux/Mac/Win). My DSP courses dealt mostly with audio signals represented as numbers in the range -1.0 to 1.0, so RtAudio is a more intuitive fit in this regard also.

On the plus side, choosing RtAudio over QtMultimedia also provided the additional ASIO support in windows ( although ASIO support is compiled and tested, I'm  defaulting to dSound on Win32 platforms for now) and RtMidi cross platform MIDI IO support ( which has also been compiled in but is waiting untapped). The STK source files have been statically incorporated into my project not because I planed it, but because I was initially only using some elements of STK... Eventually all of STK source got included. I might eventually move back to just linking to the STK library. Right now I don't care much because the code footprint is small and all the STK classes are pre-compilable, so they don't affect consequent compile times much (exception is lesson 1).

Lessons learned:

1. Complacency can make you forget the classics: Changed a define in your project file? CLEAN! REBUILD! (And don't forget the cleaning part. Too embarrassed to say how much time I wasted dealing with puzzling bugs after forgetting to do this).
2. RtAudio's closeStream() method should never be called from a destructor of a class your RtAudio instance is part of... safer to have a pointer to a RtAudio instance which you manually delete after step 3.
3. You need the "bool done" tick-data element you see in all the STK code examples... no escaping it if using the tick() callback functionality. In the main tick() callback check for data.done and return -1 if it is true; The audio engine class destructor should set the "data.done" value to true and should sleep for a couple of microseconds to allow any pending callbacks to return a -1... only then is it safe to delete the RtAudio instance without a hang (on my system at least).
4. Building STK with Qt(Qt Creator/qmake) MINGW: If like me, you added all the stk/include and stk/src files to your Qt project, you'll need to add some defines and library dependencies to your .pro file to get everything linking and compiling right. To get the compiler setting and project defines for STK, run STK's configure script from within a msys shell to generate a makefile. Reading this makefile will tell you what defines and libraries you need to add to your Qt .pro file (you can also look at the VC project files that come with STK but it is more verbose and the linker syntax is different). 

So, what do I have?

Not much really, just getting everything compiled together and first steps of simple integration for now. Tones are generating; instruments are playing; filters are filtering; effects are effecting even... but only C++ tests hacked into the audio engine's initAudio() method for the interim. The initial tests show promise. No audio script interfaces or script definable modular flow graphs have been implemented yet... that is next.

I still have a lot to do with the sound-programming before my next update... sorry, I know I can ramble sometimes. This post went on for longer than intended.

Useful links:

• http://www.mingw.org/ (msys doesn't come with Qt Creator installer)
• https://ccrma.stanford.edu/software/stk/ (STK Home page)
• http://www.music.mcgill.ca/~gary/rtaudio/ (RtAudio Home)
• http://web.mit.edu/carrien/Public/speechlab/marc_code/ADAPT_VC/rtaudio/doc/html/ (MIT RtAudio Tutorial)
• http://www.music.mcgill.ca/~gary/rtmidi/ (RtMidi Home)

What have I been doing so far:

The video shows the summary of my progress on my ECMA-based 3d engine called "ThingMaker" (for now).

It is not impressive to look at for some reasons. You're seeing a early-stage, work-in-progress debug-render of a series of tests, so there is no attempt at aesthetic appeal... The screen recorder program slowed the frame-rate down considerably... I had to compress it a lot cos my up-speed is very slow... This site's video up-loader compressed it even more... So, that's why it is so ugly.

The video starts with the engine initialized. Next I run the test script. Everything in the video is dynamically created via the script environment. There are some script-defined geometries(floating shapes), some physics objects drawn in lines(the pool table), and physics shapes with dynamically created geometries(the reddish shapes with lines around them).

The tools used to create what you see: Qt framework, Bullet Physics.

Behind the scenes:

• Script binding of basics of the Bullet API (no soft-body or constraint support yet, and only basic convex collision shapes supported).
• Script binding of OpenGL 3.3 geometries and shaders.
• Script definable collision callbacks.
• C++ interface for physical controllers, or they can be defined in script. 

...but it is still in its beginning stages. I still require rudimentary audio support in the script environment and some simple GUI elements before I can make a rudimentary game (yeah, rudimentary games with OpenGL 3.3 support, I know it sounds like a joke, but it is part of a "plan"). I left these for last since I know Qt has enough support functions to make this easier than integrating Bullet and getting OpenGL 3.3 core profile to work properly... I hope.

Much work is still needed on expanding Bullet API support and replacing the rudimentary render list with a proper scene-graph... also, only debug builds are currently maintained and the deployment strategy is somewhat flawed. I'll save that for another post entitled : "the ways in which I waste effort"...

Welcome back.

Long ago I was part of a game development house, working as the tools programmer. The work was satisfying, but I had to leave due to financial constraints. Over the past decade I have made many attempts to make games while working "real" jobs. There was always some reason why I couldn't finish making a game - time constraints, job stress, me being an idiot, lost hard drives, technology passing me by...

This February I made another attempt. It has been 9 months of in-between-projects work and I've finally typed up enough code to resemble the beginnings of a script-capable 3D game engine.

Still no games though... but they are coming... oh yes.

The intention of this blog is to share any progress I'm making and to discuss some of the thoughts behind some design issues I might need to get out in order for me to understand myself better.

Mostly though, I hope that sharing my progress will serve as motivation for myself to get the project done and start making some games!