Squashing the Wild Bug: A Behind-the-Scenes Look at Your DAW

A behind-the-scenes glimpse at what goes on inside your DAW

By Dan Goldstein

As home studio musicians, many of us know our gear intimately. With patience, we learn how to navigate our digital synthesizer's complex web of menus, and we've compiled our list of favorite go-to patches for any musical occasion. We know which reverb effects sound great and which ones sound tinny and fake. We have our MIDI and mixer channels sorted out just the way we like them. Maybe we've even opened up our guitars and wired in new pickups, or fixed a bad pot or solder connection in our vintage tube amps, or ventured inside an analog synth to re-seat a chip or replace a bad slider.

When we know how our gear works and what makes it tick, we can make better music. It's that simple. But what do we really know about our DAWs? What makes our favorite DAW tick?

We can't open it up and re-wire it. There's no technical service manual available so we can repair it when it's broken. For most of us, our DAW is a black box, an icon on our screen that can produce satisfying musical results, but which sometimes can be very frustrating. Why does it crash? Why does it work perfectly in our friend's studio, but not in ours? Why can we click the same button 50 times and everything works fine, but on the 51st push the program freezes?

As the lead designer and developer of Acoustica's Mixcraft recording software, I certainly can't explain the reason for every flaw with your DAW. But I can give you an inside look at the sorts of tools that are used to build a DAW, and the challenges that software programmers face when trying to design the perfect recording program.

TRACKING THE WILD BUG

Bugs are frustrating enough for end users, but they’re incredibly frustrating for manufacturers as well. First we have to know about them, then we have to reproduce them, and then we have to fix them . . . and hope that any bugs stay fixed if, for example, there’s an update to your operating system. I’m sure Sherlock Holmes would have related to this, as we have to collect as many clues as possible, and make logical deductions.

One bug stands out in mind as representative of one of the meanest, most resistant bugs we’ve ever encountered. This was no ordinary bug; it was extraordinary in that it was theoretically impossible. There was no way the bug could exist, nothing that Mixcraft could do to cause it, and nothing we could think of to fix it. Yet the bug persisted, and user complaints kept trickling in.

The problem was that Mixcraft users running Windows XP and loading certain plug-ins—specifically, plug-ins that used an iLok security dongle—were experiencing reboots and blue-screens when trying to load these plug-ins into Mixcraft 5. This was only happening for people using Windows XP; the same plug-ins loaded perfectly in Windows 7 and Vista. Complicating matters even further, the same plug-ins seemed to load fine in other DAWs on Windows XP. Somehow, Mixcraft was causing the problem. As you can imagine, that kind of thing doesn’t make me happy.

As we investigated the bug’s cause, we discovered that the blue-screen problem occurred when Mixcraft asked Windows to load the plug-in into memory. It was that simple: We would request that the Windows XP operating system load the plug-in into memory, and then Windows would reboot. We had no control over this. There is, in fact, no other way to load a plug-in. It was simply out of our control.

There was nothing that we could be doing that would cause the operating system to fail catastrophically when loading the plug-in into memory, yet that’s what was happening. Worse yet, other DAWs were loading the same plug-in on the same computer without any problems at all.

But then it got even more confusing because the same customers experiencing problems could load the same plug-ins perfectly into Mixcraft 4.5. That's the impossible bit. Something occurred between the end of Mixcraft 4.5 development and the beginning of Mixcraft 5.0 development that caused these plug-ins to fail, even though failure was, at least in theory, not possible.

IT’S A JUNGLE OUT THERE . . .

Perhaps the most difficult challenge when programming a DAW involves working seamlessly with audio drivers, plug-in effects, and virtual synths written by dozens of different companies, built by hundreds of different programmers with different skill levels and their own clever ideas about how to build software. There are, for example, strict specifications regarding how plug-ins should work. Steinberg, the developers of the VST specification, went to great lengths to document the VST specification to ensure that third-party plug-ins were built with uniform standards. Many programmers, naturally, view this as a challenge or perhaps an insult, and go out of their way to make sure that their plug-ins will not work (or, worse, will crash) if the rules are followed. Even Steinberg's own plug-ins sometimes violate the VST specification.

As a result, for a DAW to support a wide array of VST plug-ins, programmers must go to great lengths to ensure that these plug-ins receive the information they want in the order they require it. Certain VST features must be handled very carefully because some plug-ins will not implement these features properly. And when a plug-in does crash, which can be due to something as obvious as a bug in the plug-in, we have to go to special lengths to ensure that this crash does not crash the DAW. No one wants to lose their entire recording project simply because a plug-in stopped working.

The same is true for audio drivers. A music program that works perfectly with a thousand sound devices may fail when working with a new device whose drivers are not written properly, or which were successfully tested with a different DAW that does things in a slightly different order.

When a driver or plug-in crashes and takes down the DAW, most users will immediately assume the crash occurred because of a flaw in the DAW, which makes the manufacturer look bad. So a great deal of effort goes into ensuring compatibility with the widest range of audio drivers and software plug-ins as possible. Still, a flaw in a plug-in can corrupt system memory, return mathematically nonsensical results to the DAW, or even leave the computer's floating-point math processor in an unstable state. Any of these issues can cause the DAW to fail irrecoverably.

BEHIND THE SCENES: DIVIDE AND CONQUER

Mixcraft is written in the C++ programming language, and contains hundreds of thousands of lines of computer code, made up of millions of characters of text. In order to maintain and manage all of this code, the programming instructions are broken up and logically organized into hundreds of different files. For example, code that draws a button might be found in a file called Button.cpp. Code that resamples digital audio might be found in Resampler.cpp. This makes it much easier to maintain the code.

DAWs can involve hundreds of thousands of lines of code, all being worked on by multiple programmers.

There are several programmers working on the code at any one time, and every new feature means making changes to potentially dozens of different files, and adding many new files in the process. And there's a twist: I do most of my programming from my home studio near Las Vegas, Nevada, while the rest of our Mixcraft developers are located in California. Given all of that, how are programmers in different geographical locations able to develop the same program at the same time, while making changes to the same files?

The answer is surprisingly easy: serious programming teams store their code in a repository. This is a server on the internet that acts as a central database for all of the files that make up a program. Think of it as a library. When I want to make changes, I "check out" the code, make my changes, then "check in" the changes. At any time I can "check out" the latest changes to the code to see what other improvements my team has "checked in." In the rare instance where multiple programmers have made changes to the same text in the same file, the Code Repository will detect a conflict, and the programmer will have to figure out what the correct code should be.

Part of the beauty of this system is that it's not just a storage bin for code. The repository also acts as a revision control system for the code. That may sound complicated, but it's not—it simply means that the database remembers every change that was ever checked in to the repository. So at any time it's possible to travel back in time, so to speak, and see what the code looked like on any given date. That makes it much easier to find what went wrong, when, and why.

MISSION IMPOSSIBLE

In the case of this “impossible” bug, however, we found ourselves with a dilemma. A blue-screen error or system reboot simply cannot be caused by a software glitch. These problems occur deep at the driver level, and are caused by a communication failure between the hardware and the operating system. While it was evident that the blue-screen was happening because of the plug-in's use of the iLok hardware, we were still completely puzzled: how could Mixcraft's code possibly interfere with the plug-ins ability to talk to the iLok device? And what could possibly have changed between Mixcraft 4.5 and Mixcraft 5 that could cause this problem?

Who would have thought that this, coupled with an obscure issue involving a 10-year-old operating system, could bring down a DAW?

We had a Windows XP computer and a copy of East West Symphonic Orchestra Silver Edition (a fantastic plug-in that uses the iLok system), which we could use to reproduce the problem. We knew that Mixcraft 4.5 did not have the bug. And we knew that we could use our code repository's revision control system to see what the code looked like on any given date. So we began looking for clues.

I checked out the state of the code on May 1st, 2009, a date early in the development of Mixcraft 5. This code had very little changed from the final release of Mixcraft 4.5. When I compiled this code and ran it on the Windows XP machine, I discovered that the code was able to load the plug-in without crashing. So, we had a good starting point.

I then checked out the code as it existed on July 1st, 2009. At this point in the development, effects automation had been added. This code, it turned out, was still able to load the plug-in without fail. So, I skipped ahead to September 15th, 2009. Mixcraft 5's mixer interface had been added, along with improvements to the audio pan control. When I tried to load the plug-in with this version of the code, the computer crashed. Aha! Something changed between July 1st the September 15th that caused the crash to occur—but what?

I went back a month, to August 15th, 2009. Basic support for loading video files onto a video track had been added to the code at this point in time. I tested the East West plug-in—and it caused a system crash. Could the addition of video features somehow be causing the crash? It was certainly a possibility, though I still couldn't fathom how.

Next step was setting the time machine for August 1st, 2009. Some preparation for the video track had been done, but there were no video features in Mixcraft yet. The plug-in still crashed, though. That meant that video could not be the culprit. I now knew that whatever was going wrong had been introduced between July 1st and August 1st, so the bug’s introduction was now narrowed down to a one-month time period.

I went back one week further, to July 25th, 2009. I compiled the code, and the plug-in was able to load without crashing the computer. That gave a one week window, which made it much easier to focus in on the problem.

SQUASHING THE BUG

In the end, I looked at the changes that had been checked in to the repository between July 25th and August 1st. Most of these changes were minor, incremental changes, but one major feature was added that week: A convenient CPU meter, located in Mixcraft’s lower right corner.

The CPU meter works by periodically talking to the operating system to get measurements on the amount of work being done by the processors. This involves the software talking to Windows, and probably involves Windows talking to the processor or processors that control the computer. For some reason, the conflict between the iLok hardware and Windows XP occurred when performing a CPU check while loading an iLok-protected plug-in.

I couldn't begin to tell you exactly why this happens—it's just some minor, hidden flaw in a decade-old version of Windows that we happened to expose. It shouldn't have been possible for Mixcraft to cause a blue-screen driver error, but our actions were in fact indirectly responsible for the problem. There was no way to fix the error other than to remove the CPU meter from Mixcraft for users running Windows XP, but once we did this, Mixcraft 5 suddenly became fully compatible with all of the iLok-hungry high-end plug-ins that our Windows XP customers were trying to use. No longer were they forced to use an old version of Mixcraft just to run their favorite plug-ins.

Here's the mug shot of the iLok's accomplice.

It was a small victory, but at the same time, an important one for those customers that this bug inconvenienced. Without the use of our code repository's revision history, it would have been nearly impossible to discover what caused the problem. Mixcraft is designed as a professional DAW with hundreds of features; who would ever think that the little CPU meter could cause such a catastrophic system failure? We certainly didn't.

When we've done our job right, and written our software to handle every audio driver and effect plug-in and virtual synthesizer and operating system and computer processor that you can throw at it, Mixcraft will just run. You'll make your music and have fun, and honestly, that's exactly the way it should be.

But it's nice, for a moment, for us to be able to tell you that all these tools—the compilers, the repositories, the test computers, the closet full of audio hardware and CDs full of drivers, the piles of plug-ins—all of these tools are here in our lab so that we can find and fix the little frustrating issues that can pop up in software. No software is perfect, of course, and new bugs will always appear—it's an unfortunate fact of life. And when these bugs are caused by poorly-written plug-ins or badly designed drivers, we simply can't fix them—although we can alert companies to any problems we find.

But sometimes, with some detective work and the right tools, even the most impossible bug can be caught and squished. I hope you enjoyed getting an inside look at what goes on inside the software development process, and may your system be free of bugs—especially the “impossible” kind!

Dan Goldstein is the lead developer of Acoustica's award-winning Mixcraft recording software for Windows. You can download a free trial of the software at www.acoustica.com/mixcraft