DAW TEST

[Microstudio]

I wanted to do a DAW TEST for myself and thought I would post it for all. After you listen vote. All details are on a new page I made on mysite.

Take the DAW TEST there is no right answer you just hear what you hear.

Rock On!

[TotalSonic]

Your test is seriously flawed in one major way:
it is not blind. i.e. you should label the choices A, B, C - or 1, 2, 3 - or similar (i.e. X, Y, Z) - and not tell people what they are choosing.

Otherwise whatever presumptions and biases they have going into the test will highly effect their choices. As the test exists now I'd say whatever the poll results come to are fairly useless in terms of determining anything except what bias exists in the poll takers.

I haven't downloaded your files yet but can you confirm you did the following:
* either used the same exact recording that had been previously recorded and then rendered this in all 3 DAW's - or split the exact same signal chain of the exact same performance so it was recorded to all DAW's simultaneously? Otherwise people will be judging the performance differences and not the differences between a DAW's rendering of a single performance.

* level matched all the mixes to within 0.1dB of each other? Otherwise whatever mix is even a tiny bit louder will be perceived as "better."

Sorry to be a stickler in this regard - but "tests" of audio where subtle differences need to be detected are very easily invalidated unless some VERY strong psychological factors which can easily skew results are eliminated as best as possible.

Best regards,
Steve Berson

[Grekim]

Originally Posted by TotalSonic

Your test is seriously flawed in one major way:
it is not blind. i.e. you should label the choices A, B, C - or 1, 2, 3 - or similar (i.e. X, Y, Z) - and not tell people what they are choosing.

Otherwise whatever presumptions and biases they have going into the test will highly effect their choices. As the test exists now I'd say whatever the poll results come to are fairly useless in terms of determining anything except what bias exists in the poll takers.

I haven't downloaded your files yet but can you confirm you did the following:
* either used the same exact recording that had been previously recorded and then rendered this in all 3 DAW's - or split the exact same signal chain of the exact same performance so it was recorded to all DAW's simultaneously? Otherwise people will be judging the performance differences and not the differences between a DAW's rendering of a single performance.

* level matched all the mixes to within 0.1dB of each other? Otherwise whatever mix is even a tiny bit louder will be perceived as "better."

Sorry to be a stickler in this regard - but "tests" of audio where subtle differences need to be detected are very easily invalidated unless some VERY strong psychological factors which can easily skew results are eliminated as best as possible.

Best regards,
Steve Berson

+1. I didn't want to be the first to rain upon your fun. If you set up the test right, as Steve outlined, then you shouldn't hear any difference unless somehow one of the DAW's interacts worse with the audio card's drivers.
Make things more intersting by taking the same track and processing them with fx unique to the different DAWs. Get settings as close as possible. Make things more intersting by doing this with a full mix. Obviously, this would be incredibly difficult to do.
Just be glad SAW sounds great and doesn't cave in with a giant mix!

[Microstudio]

Originally Posted by TotalSonic

Your test is seriously flawed in one major way:
it is not blind. i.e. you should label the choices A, B, C - or 1, 2, 3 - or similar (i.e. X, Y, Z) - and not tell people what they are choosing.

Otherwise whatever presumptions and biases they have going into the test will highly effect their choices. As the test exists now I'd say whatever the poll results come to are fairly useless in terms of determining anything except what bias exists in the poll takers.

I haven't downloaded your files yet but can you confirm you did the following:
* split the exact same signal chain of the exact same performance so it was recorded to all DAW's simultaneously? Otherwise people will be judging the performance differences and not the differences between a DAW's rendering of a single performance.

* level matched all the mixes to within 0.1dB of each other? Otherwise whatever mix is even a tiny bit louder will be perceived as "better."

Sorry to be a stickler in this regard - but "tests" of audio where subtle differences need to be detected are very easily invalidated unless some VERY strong psychological factors which can easily skew results are eliminated as best as possible.

Best regards,
Steve Berson

There are a ton of ways you can do a test. I did not make it blind becuase in the end you will have to tell people and even then it can effect them other then the truth of what they hear. You could have a 100 or more people in the room and see how the test was done and some will so they don't think its fair.

I opened each DAW and recorded the same way. Me playing a simple cord progression 3 times the same way to the best of my ability, I have been playing for over 30 years and understand about playing consitently and the levels will show this I hope.

I do not have 3 identical PC's, audio cards and iloks to record at the same time.

I have never seen a DAW test that someone found to be not fair... o well I did what I did and I know that it was done fair, I own and use all 3 DAW's and "I" wanted to see if there was a difference so I did the test for myself and thought I would share it. Simple

[Microstudio]

Originally Posted by Grekim

+1. I didn't want to be the first to rain upon your fun. If you set up the test right, as Steve outlined, then you shouldn't hear any difference unless somehow one of the DAW's interacts worse with the audio card's drivers.

Thats part of the test I guess drivers have a lot to do with DAW preformance and one would think that m-audio has pretty good drivers that work with most all DAW's in fact all three DAW's work with Asio drivers and thats what was used.

[TotalSonic]

Originally Posted by Microstudio

Thats part of the test I guess drivers have a lot to do with DAW preformance and one would think that m-audio has pretty good drivers that work with most all DAW's in fact all three DAW's work with Asio drivers and thats what was used.

Drivers will have the effect on usable latencies and on how many tracks can be recorded simultaneously, and these two factors can indeed vary with the DAW app based on how efficient it is as well - but unless something is seriously wrong (i.e. "broken" software or drivers), or if you are running buffers so low as to have actual glitching - then recording quality should be related only to the quality of the AD converters and not to what DAW is used (as long as no further processing is done). This can be verified by splitting the signal digitally after the ADC (easy to do with a digital patch bay like the Z-sys) and recording into DAW's from the same stream simultaneously - and then doing a null test between the two results.

To effect a null test:
line up the two files all the way to single sample accuracy on two different tracks starting at the same point, and then polarity is reversed on the 2nd track, and then mix these two files together. Whatever difference between the two files will show in the resulting rendered mix.

Test tones (to make sure the files are level matched) and a time alignment pulse (to make lining the files up with sample accuracy) sent at the head of the recording can make the null test easier to perform.

Best regards,
Steve Berson

[Tim Miskimon]

Looking at the meters & the wave forms show that all three wav files have differant levels so I'm not sure the test would be accurate.
On playback the Propellerhead record file is louder than both the Saw & Protools mix.
That said I think the overall tone of the Saw recording is both fuller & cleaner sounding.
I've found this to be true in the compare tests I've done of my own over the past 10 years.
That's one of the main reasons I'm still using Saw after 16 years.
So many things come into play besides the software - converters, mics, etc. so I stopped doing these kinds of tests a few years ago.
Most of the time I hear the difference is when I've done a project in Saw & for what ever reason had to finish or add something to it at another studio using Nuendo, Pro Tools or some other software.
I could hear slight differences in tones & stuff so that's pretty much how I've come to my conclusions.
I believe one should use what ever tool that works for them - afterall in the end it's the song & performance that really matters and I don't think most listeners will notice these differences nor do I think they care how something was recorded or what equipment was used.

[TotalSonic]

Originally Posted by Microstudio

There are a ton of ways you can do a test. I did not make it blind becuase in the end you will have to tell people and even then it can effect them other then the truth of what they hear.

That's absurd. You can easily run the poll for a set time with the options posted blind - and then you close the poll and show the results.

You could have a 100 or more people in the room and see how the test was done and some will so they don't think its fair.

True - but as your test exists now it's just plain useless. Please don't take this personally - but either scientific process is applied or we're just flinging garbage statistics around.

I opened each DAW and recorded the same way. Me playing a simple cord progression 3 times the same way to the best of my ability, I have been playing for over 30 years and understand about playing consitently and the levels will show this I hope.

Nope - sorry then the test is completely invalid. Regardless of your 30 years experience you can easily move a few millimeters (or likely more) during the recording - or have your strum be slightly different each time - and you will in fact effect a performance slightly different that will in fact completely negate the ability to hear what may or may not be different between the 3 DAW's. Again - while trying to detect subtle differences between a recording it's absolutely vital to eliminate differences in performance.

I do not have 3 identical PC's, audio cards and iloks to record at the same time.

So instead just record a single performance to a single other app and then render a mix in the 3 DAW's. This would eliminate people judging the performance rather than the DAW.

I have never seen a DAW test that someone found to be not fair...

However there are tests that the vast majority can agree have done everything possible to follow the basic principles of scientific experimentation. Your test currently follows none of them.

o well I did what I did and I know that it was done fair, I own and use all 3 DAW's and "I" wanted to see if there was a difference so I did the test for myself and thought I would share it. Simple

That's all good. However - you can easily be misleading yourself as well.

Again - if you really wanted to test this you need to eliminate your own bias by first having the exact same performance played back to with someone else playing you back the different choices, and without you knowing which source you are listening to. There are a number of softwares that offer automated A/B/X testing as well - this could also be helpful.

Best regards,
Steve Berson

[Microstudio]

I did line up all 3 files in SAW and looked at the levels and Record and SAW have pretty much the same levels and PT was the lowest but for the most part the levels show I played all 3 takes at a similar level and I can tell you after using all 3 for months this test does reflect the level differences each DAW does have when recording for me with my equipment.

[TotalSonic]

Originally Posted by Tim Miskimon

Looking at the meters & the wave forms show that all three wav files have differant levels so I'm not sure the test would be accurate.
On playback the Propellerhead record file is louder than both the Saw & Protools mix.
That said I think the overall tone of the Saw recording is both fuller & cleaner sounding.

Again - each file is a different performance. So there's no way to determine whether the "fuller & cleaner" is a result of the DAW or the result of the performance. I'd likely pick the latter.

I've found this to be true in the compare tests I've done of my own over the past 10 years.
That's one of the main reasons I'm still using Saw after 16 years.
So many things come into play besides the software - converters, mics, etc. so I stopped doing these kinds of tests a few years ago.
Most of the time I hear the difference is when I've done a project in Saw & for what ever reason had to finish or add something to it at another studio using Nuendo, Pro Tools or some other software.
I could hear slight differences in tones & stuff so that's pretty much how I've come to my conclusions.
I believe one should use what ever tool that works for them - afterall in the end it's the song & performance that really matters and I don't think most listeners will notice these differences nor do I think they care how something was recorded or what equipment was used.

Where I think differences are actually perceivable is after fully processing a mix - this includes panning, equalization, dynamic processing, etc.

Best regards,
Steve Berson