Stability of SAC on my system [Archive] - SAWStudio / SAC User Group

Tony Marone

12-10-2008, 07:56 AM

Well here I am again! You may recall in a previous post that I was having lock-up problems with SAC on the system I had building. I think I found the culprit - the motherboard. It had an nvidia chipset on it and I believe that was the problem. I now am using an Asus P5Q as Jesse posted in his system description and everything seems fine - - - but - - - now I have a new problem. SAC will run for a 24 to 30 hour period just fine with no dropped buffers, perfect audio, and no other problems at about 4 to 5% load then out of the blue, the CHANNEL METERS will freeze and the Windows message "SAC.exe has generated errors and needs to close" appears. Audio will continue, you can move the sliders up and down and they will still control the output, even the compressor meter still works. Maybe no one has noticed this cause they haven't been as thorough at burning in as this, or maybe once again its something wrong with my system. I have ran this test setup 3 times over 5 days and had the same result each time. What could be going on?:D

Thanks again,
Tony Marone

Bob L

12-10-2008, 09:01 AM

Any chance of some background service kicking in every 24 - 30 hours?

Some kind of auto system backup? How about the System Restore automatic system checkpoints? You might try turning OFF system restore and let it run for 30 hours again.

When this happens, can you still shutdown and exit the program... or do you have to reboot?

Also... what version are you testing? Make sure you are using the current version 1.3... there have been lots of fixes already... and more to come.

Bob L

Naturally Digital

12-10-2008, 11:32 AM

Maybe no one has noticed this cause they haven't been as thorough at burning in as this, or maybe once again its something wrong with my system. I have ran this test setup 3 times over 5 days and had the same result each time. What could be going on?Tony, I reported the same problem to Bob a few weeks ago. I'm running it on a P5W-DH Deluxe. It's repeatable on my system and crashes after the same time period as you (24-30hrs).

None of the suggestions apply to my system (System Restore is off, no background backups, latest version etc. Nothing I can think of that would cause this) and I've been running the SAC buffers at 4x1024 with the DWAVE driver model.

At this point I would like to know how many people *don't* have this issue. This has never happened while I was operating the DAW. It is when I leave it alone (I have a KVM and four machines connected) for long periods of time but with SAC running.

I'm sure we'll get to the bottom of this and resolve it soon enough. We just need enough information to track it down.

Bob L

12-10-2008, 01:33 PM

I guess I will have to start leaving a few SAC systems running fulltime in the lab and see if this is something that they all exhibit.

Not that it should cause any immediate problems using SAC in a live show environment... I'm sure not too many of you are doing shows that run straight for 24 hours without a shutdown. :)

Bob L

Warren

12-10-2008, 03:05 PM

Tony, I reported the same problem to Bob a few weeks ago. I'm running it on a P5W-DH Deluxe. It's repeatable on my system and crashes after the same time period as you (24-30hrs).

None of the suggestions apply to my system (System Restore is off, no background backups, latest version etc. Nothing I can think of that would cause this) and I've been running the SAC buffers at 4x1024 with the DWAVE driver model.

At this point I would like to know how many people *don't* have this issue. This has never happened while I was operating the DAW. It is when I leave it alone (I have a KVM and four machines connected) for long periods of time but with SAC running.

I'm sure we'll get to the bottom of this and resolve it soon enough. We just need enough information to track it down.

Dave are you using the Mykerinos audio cards to get a DWAVE driver to run in SAW/SAC? Also what card manufactures currently support DWAVE

Warren

Naturally Digital

12-10-2008, 03:14 PM

Dave are you using the Mykerinos audio cards to get a DWAVE driver to run in SAW/SAC? Also what card manufactures currently support DWAVENo, I'm running a Soundscape Mixpander/9 card. AFAIK only the Soundscape/Sydec/SSL cards support the DWAVE driver.

Bob L

12-10-2008, 04:03 PM

Warren,

Try using the ASIO driver and see what happens... the current DWave driver is a very old implementation and could be the source of the problem.

Bob L

Warren

12-10-2008, 04:28 PM

Warren,

Try using the ASIO driver and see what happens... the current DWave driver is a very old implementation and could be the source of the problem.

Bob L

Sorry Bob:
I did not mean to confuse the issue, I don't use DWAVE drivers nor am I having any issues (other than whats between my ears) I just wanted to know what card Dave was using that supported DWAVE. But once again your right on top of all our concerns, thanks again for the #1 rated consumer support!

Chris

Warren

12-10-2008, 04:31 PM

No, I'm running a Soundscape Mixpander/9 card. AFAIK only the Soundscape/Sydec/SSL cards support the DWAVE driver.

Thanks Dave: good info!
? is not that quite high fo a buffer setting? or is that for SAW during playback with a lot of plugs?

Chris

Bob L

12-10-2008, 05:21 PM

Sorry... I guess that message was meant for Dave, and/or Tony.

Bob L

Jesse

12-10-2008, 07:48 PM

Tony

i've never come to my mind to do that kind of test of 24-30 hours running SAC unless you're doing a 3 days festival or something :D, the maximum time i end up it's 5-6 hours straight.

In many ocassions instead of using the mute buttons i stop SAC engine between breaks to avoid someone talk on the mics when my band it's resting or when a DJ it's performing using his own equipment.

In that way it's like giving SAC a "break" too, but even if all of us do a festival of 3 days i think always there's time to do a SAC shutdown, i think we could close SAC and running it again in less than 30 seconds.

Just my opinion :)

Jesse

Bob L

12-10-2008, 07:54 PM

You can simply recycle the engine in a few ms by using the Shift-Left-Click on the Live Lite.

In fact you could do this in the middle of a song and I doubt the audience would ever hear it. :)

Bob L

Jesse

12-10-2008, 08:07 PM

Wooo :eek: Bob you're the man, i didn't know that :)

Well, that's easier than my solution :D

Tony Marone

12-11-2008, 03:43 AM

Bob - I don't think the engine is the problem, it seems to be doing just fine:D, still playing through the fire! I love what you are doing here, this is very cool software. I think there must be something odd going on with the channel meters.

Jesse - Of course we want the system to run 24/7/365 without as much as a glitch!:). After all, knowing there are no gremlins lurking can give you the reassurance to run 24 channels with a band and god knows what else in front of a few thousand peeps. If things fail then, who they gonna be lookin' at?

All - I have a broadcast automation system called 'The Phantom' from Register Data Systems that ran 24/7/365 for 5 years and 8 months finally to fail because the hard drive ground to a halt. It is on a UPS, runs under DOS 6.22 on a Pentium 90, and is what I call rock solid. It is still running today with about 4 years on the current drive.

Thanks all,
Tony Marone

Bob L

12-11-2008, 04:41 AM

I don't think you ever answered my question about what happens after the meters freeze... you say the engine is still running... but... do you still have control after that... can you stop and restart the engine... can you dtill control the rest of the program... can you exit properly... if you stop and restart the engine do the meters clear and start working again?

Bob L

Tony Marone

12-11-2008, 06:23 AM

Hi Bob,

You can still control volume of channels with sliders - once you turn the engine off that is it, no restarting it again:(, you have to bail out through the 'has generated errors' window and then you are back at a normal empty desktop from which you can start up SAC again.

Hope this helps,
Tony

Bob L

12-11-2008, 07:00 AM

That sure sounds like the soundcard driver has locked up... next time try using the Force Engine Shutdown {Break} key... see if that is able to grab control of the driver and reset it.

Bob L

DominicPerry

12-11-2008, 07:57 AM

Here's my guess, based on the interval you are describing - 24-30 hours - I think something is trying to update, maybe Windows itself (2 services to disable), Adobe Acrobat, something sitting on Task Scheduler (I always have the Task Scheduler service disabled), anti-virus (even if you have AV disabled, it will probably still update regularly), windows time update (usually every 7 days but may be more often if it can't 'get through' - disable the service), backup scheduler (I think this was mentioned already).

It's hard to clean windows up completely.

Dominic

Tony Marone

12-11-2008, 09:12 AM

Bob - Then why do I still have audio and mixer control of said audio?

Dominic - Fresh install of WinXP sp2 with nothing else installed on the box but SAC and the RME driver - only 14 processes running - I don't think that is the answer.

Appears that something having to do with meters may be corrupting important memory over time or writing to memory that it shouldn't. I programmed an application exclusively in ASM so I've seen it happen before, maybe even been responsible for it myself:o. These things can be hard to find and sometimes only occur when the code is stressed for a length of time.

Thanks,
Tony

Bob L

12-11-2008, 09:25 AM

Anything is possible... but the other suggestions sound more likely.

What soundcard were you using Tony? Firewire by chance?

Bob L

Warren

12-11-2008, 10:46 AM

A thought:

Do you have windows Internet time enabled?

Double click on the clock bottom right hand corner of screen
then click on internet time and disable "automatically sync with internet time server" I am not sure this is it but it may try to connect every 24 hours. Worth a try.

Chris

IraSeigel

12-11-2008, 11:05 AM

Naturally Digital

12-11-2008, 12:16 PM

but the other suggestions sound more likely.None of these other suggestions explain whay "SAC has generated errors and must be shut down". We're not talking about glitching the audio here, we're talking about a crash of SAC. Even if an automatic update or some other windows service is causing a problem for SAC, it should not crash hard.

My current application for SAC is in the studio and I want to be able to leave it running for days on end, just like my hardware pieces and the same as I do with my Soundscape mixer software. If Tony and I are the only ones with this problem then obviously it's something in our setups. Unfortunately we can't know that until some other people try running SAC for a few days at a time.

I've checked my system logs and the crash seems repeatable and predictable.

It's not a deal-breaker for me because it hasn't happened during a session and I don't *need* to leave it running for days... I just don't want to see us ignore a possible bug for longer than necessary.

I'm more than willing to help by trying to track it down to something on my system. I'll supply specs, settings, mix files, lists of plugins... I'll try clean mix files, specific combinations, ASIO drivers... whatever it takes. I don't see a big rush to fix this (for me anyway) but the first step would seem to be checking if the same problem exists on a different setup. I can do that myself, I just haven't had time to do a SAC install and setup on another machine yet.

I guess I will have to start leaving a few SAC systems running fulltime in the lab and see if this is something that they all exhibit.

I'm looking forward to the results of those tests. Thank you Bob!

Bob L

12-11-2008, 01:39 PM

I doubt that SAC itself is crashing... but from the sound of the symptoms... it's more likely the soundcard driver getting stepped on... which would then translate into an error message from SAC when it next attempts to communicate with the driver for more data.

Even the lowest level thread can be running automatically and still feeding data to the driver (the continued audio playback)... but higher level communication functions could be crashed.

Again... this is all theory... obviously there is not enough info yet to chase this one down... and I will have more to go on if I can duplicate the crash here after a 24-30 hour run.

Bob L

DominicPerry

12-11-2008, 01:55 PM

Naturally Digital

12-11-2008, 02:05 PM

What do I need to know to do this test Tony/Dave? I can run it on my laptop - what are the channels, setups, networking, scenes, number of monitor mixes, etc you are using? Is this SAC alone or feeding SAW?In my case I've had SAW feeding SAC (but not in playback mode, just stopped) with 10 channels and 3 mixes in SAC (FOH, Mon 1 and Mon 2). SAC is latched in Live mode. I have the Sonalksis mkII compressor on the FOH channels, the Sonoris Leveler on Mon 1 channels and the Kjaerhus GCO on Mon 2 channels. Some of these mixes get sub-mixed to group outs before the final output. I've been working on the same project on and off for a few weeks so this has not changed much.

I would suggest to just start with a clean mix file, set up some ins and outs and let it run for a couple days. I'll be doing a similar test shortly. If I can't duplicate the crash with a clean mix file then I'll start chasing down the plugins.

DominicPerry

12-11-2008, 02:58 PM

In my case I've had SAW feeding SAC (but not in playback mode, just stopped) with 10 channels and 3 mixes in SAC (FOH, Mon 1 and Mon 2). SAC is latched in Live mode. I have the Sonalksis mkII compressor on the FOH channels, the Sonoris Leveler on Mon 1 channels and the Kjaerhus GCO on Mon 2 channels. Some of these mixes get sub-mixed to group outs before the final output. I've been working on the same project on and off for a few weeks so this has not changed much.

I would suggest to just start with a clean mix file, set up some ins and outs and let it run for a couple days. I'll be doing a similar test shortly. If I can't duplicate the crash with a clean mix file then I'll start chasing down the plugins.

Thanks Dave. OK, clean it is. I think we know what's going to happen though!

Dominic

Tony Marone

12-11-2008, 04:07 PM

Bob - Soundcard is RME Hammerfall DIGI9636 PCI, ASIO driver, 64,2. Once again, although I don't know the specifics of your programming, it seems unlikely to me that I would still have controllable audio coming out of the system if the sound processing loop were corrupted (sound driver stepped on).

When I had this card in the Dell Optiplex before, it was my main computer that I did everything not audio on. Even with me running all kinds of things like VLC media player playing pron video;), photoshop open, Quartus II open editing an FPGA file and using its USB programmer output, and a very graphics intensive circuit board layout using PCAD all running at the same time as SAC with 16 channels, eq, compressors, gates, I had no more than 2 or 3 slipped output buffers in my 5 hours of testing. ROCK freakin' SOLID, and this machine has all the usual offenders on! but the test was only 5 hours . . .

I really think this is time related and a slow leak that may have been there all along. I will be happy to be proven wrong, but I think you do yourself a great dis-service by thinking your real time processing loop could be so unstable! If we have problems in that area, it usually shows up much sooner than later:)!

Bob L

12-11-2008, 05:41 PM

The engine has proven to be pretty darn solid... what I am thinking now is something about an internal sample counter which controls the meter timing maxing out after that much time... and reseting to zero which may then be confusing the meters...

But if that were the case... it confuses me why you could not control and stop the engine cleanly... the meters are not connected directly to the driver in that manner... so if the problem is not driver related, then the engine should have been able to be stopped correctly and then restarted, which would have reset all internal counters.

And, by the way... yes... a frozen driver can still happily be outputting audio at the hardware level while its main interface connection has been crashed at the app level... my audio loop is handled at the highest thread priority the system offers and could easily keep going while another part of the interface is crashed.

Bob L

Tony Marone

12-12-2008, 02:11 AM

Bob,

The engine can be stopped normally, but will not restart.

Also, as I said before, I still have control of audio levels through individual channel sliders and output sliders. And the gain reduction meter is still working. Wouldn't that mean that the driver was still working?

Or maybe all audio mixing is done in software and the audio card is just putting out the resulting 'software' mixdown:o duh.

Well maybe another clue is that when I go from I to O and back, the input meter that was frozen has disappeared (no meter reading at all) as if it is no longer being updated at all even with audio still playing and controllable with the slider.

Tony

Bob L

12-12-2008, 08:35 AM

Well... I am checking it now... I left SAC in my main production rig running since yesterday at about 3pm... so I will reach the 24 hour mark this afternoon... I'll see what happens.

Bob L

Naturally Digital

12-12-2008, 11:07 AM

Well... I am checking it now... I left SAC in my main production rig running since yesterday at about 3pm... so I will reach the 24 hour mark this afternoon... I'll see what happens.Mine crashed again last night sometime. I don't get to see it crash because I've always discovered it when I KVM over to that machine and there's the windows error report (send/don't send) message. Personally I've never had the chance to check if SAC is still responding because when I click 'don't send', SAC closes.

Anyway, it may be longer than 24hrs but during the time I had SAC running I operated SAW and SAC, playing back an edl maybe two or three times (during the 24-30hr period). I just never closed and re-opened SAC and I think I had the SAC engine running (green light on) the whole time.

Bob L

12-12-2008, 11:37 AM

In another test I just did in my debugger, one of my suspicians proved to be confirmed... and I was able to create the meter freeze x number of seconds or mins after starting the engine by fudging certain counter variables to startout as if they have been running for x number of hours.

But... as I also suspected, after the meter freeze and the blank meter activity once switching to the output chans and then back to the inputs... exactly as Tony has observed... a simple right-click on the Live Lite to stop the engine and another right-click to start it again and the problem was reset for another 24 hours.

So... no lockup of the engine as Tony seems to mention... I also was able to instantly reset everything by simply Shift-Left-Clicking the Live Lite to recycle and reset the engine instantly... so..

Tony... are we clear about your engine not stopping and restarting by right-clicking the live lite? Is that what you tried... because it sure appears to me that I've found the bug... other than explaining why the system is locked from restarting in your experience.

Although... I am still waiting to see the crash for real on my other system that has been running all night... perhaps something else gets invloved in the real 24 hour situation rather than the simulation I am doing in this test.

Anyway... I'm on it... and this should be flushed out soon enough.

Good detective work... this is one I probably would never have seen, because I would never have thought to leave the engine active for that long without some start and stop happening in between.

Bob L

IraSeigel

12-12-2008, 11:42 AM

Bob,
I think Tom Hanks should play you in "The Lentini Code".

I'm excited to hear of your findings.

Ira

Bob L

12-12-2008, 12:30 PM

Well... it appears my meter fix works fine on my simulated 24 hour debug code... now we'll have to wait until my production rig reaches its 24 hour limit and see if the problem also involves some other counters and pointers that will cripple the engine reset.

Bob L

DominicPerry

12-12-2008, 12:34 PM

Running one here too, but I have another 3 hours to reach the 24 hour marker.

Dominic

Bob L

12-12-2008, 03:14 PM

Ok... my system reached its mark at about 23 hours... yup... crash.... ok... so null-out the previous discussions about your system... and let me dive further in... I definitely fixed the meter part... but there might be some other engine details that need resetting when the counters roll-over also.

Although now I will have to retest this new version for the next 24 hours and see if in fact the meter fix was also causing the crash after a real rollover period.

Bob L

IraSeigel

12-12-2008, 05:12 PM

...

Although now I will have to retest this new version for the next 24 hours and see if in fact the meter fix was also causing the crash after a real rollover period.

Bob L

Has anyone tried "fast forwarding" their computer clock, or is this something to do with the actual passing of time? Sorry, I don't know how to ask the question properly.

Ira

Warren

12-12-2008, 05:28 PM

Has anyone tried "fast forwarding" their computer clock, or is this something to do with the actual passing of time? Sorry, I don't know how to ask the question properly.

Ira

I think that fault may come in to place only if the internet time server had anything to do with the crash. Maybe, could be, heck I don't know. Most assuredly.

Chris

Bob L

12-12-2008, 05:46 PM

No... my observation is that is has to do with my internal variables counting up... and growing larger and larger till they actually reach their max value (based on variable size) and then reset to zero and start up again... the reset to zero is where the problem occurs... as the engine expects certain variables to be in front of others... then all of a sudden... they are behind... very tricky stuff indeed... and the max counts have never really been reached in SS, since most people have not left it running for 24 hours straight without stopping the playback... oh well... SAC is playing in new territory. :)

Bob L

Rick Stansby

12-12-2008, 06:24 PM

I just read this thread, and I have to say that Bob is an amazing guy for attacking this problem so quickly, and for figuring out what was going on. Most any other company would tell you to stop running the program for 24 hours straight, or blame a third party.

DominicPerry

12-12-2008, 06:37 PM

Mine just crashed - 27 hours. Is there anything I can do/send/test to help or do you have this covered now Bob?

Dominic

DominicPerry

12-12-2008, 06:39 PM

The sliders still work and the sound still comes out. But the crash report needs sending!

Just 6 duplicate channels, using the internal sound card with EQ and Comp, nothing else, all going to one stereo out.

No plugins, a fresh install of SAC 1.3

Dominic

Bob L

12-12-2008, 08:00 PM

Yup... the problem is definitely some of my internal variable counters... obviously, the engine itself is stable, until those counters wrap around... the time it takes depends on your buffer size and sample rate... but in general, its way up there near the 24 hour mark, and should not interfere with normal show use... and in the studio or a permanent installation, all that's needed is to stop and restart the engine within that 24 hour period to reset everything.

I am watching my rig again with some debug display code displaying some of the important counters so I can see which ones might be causing the problem... but... unfortunately... it looks like I have to wait the full 24 hour period so everything is expanded out for real, rather than use my faked test that I mentioned earlier. :(

Bob L

Tony Marone

12-13-2008, 10:56 PM

Well Bob?:p

(Theme from Jeopardy plays in background for the 93d time)

Inquiring minds want to know! Did yours crash again? Do you have the answer yet? We have all the confidence in the world - let us know how its coming.

Tony

Bob L

12-13-2008, 11:35 PM

Yes... crash at 27 hours... the variables I altered and were displaying were not the problem for the crash... they did fix the meter freeze... but not the engine freeze... so... there are other engine variables that are involved that need to be reset at the wrap-around point also...

Oh well... have to dig through the code... there are only tens of thousands of variables involved so it shouldn't be too dificult. :(

The drag is that, the engine will have to run for the full 27 hours again to find out if its fixed... this may not be fun!

Bob L

Tony Marone

12-14-2008, 01:20 AM

Bob

Just asking some questions out of curiosity and maybe to get you thinking:)

Did you ever consider that the variables could wrap around in use and what would happen if they did?

Could you add extra bits to avoid a wraparound altogether?

Could you orchestrate the wraparound by adjusting the length of some variables in relation to others so that the wraparound would not be critical?

Could you preset what you deem to be the critical variables to near wraparound so that you wouldn't have such a long wait time during testing?

Not trying to be too nosy, but being an ex-assembly programmer myself with a commercial product out there for 16 years, I find this a fascinating problem.

Tony

Bob L

12-14-2008, 08:41 AM

Well... its obvious now that this IS a wrap-around problem... and adding bits will not solve the eventual fact that those variables will still wrap... only with a longer time interval... and it's also obvious that will never satisfy those who decided that this was a problem in the first place...

I can hear the arguments now that SAC will need to run for 27 years without stopping in such and such an installation... and if it doen't, then it can't be used. :)

So... I am off to find the culprits... buried deep within the engine...

Bob L

Iain Westland

12-14-2008, 09:19 AM

i would not put it high on your list though, if its stable at 16 hours thats long enough for us, longer would be nice but only cos i like to have the most stable DAW in't world

Iain

Tony Marone

12-14-2008, 09:40 AM

Bob,

You could just make a new counter to count up to lets say 20 hours or whatever you think would be the shortest period that would be failure proof and then do an automatic reset of the engine, thereby avoiding the problem altogether. :) I think you said the reset is almost undetectable.

That might be a stop-gap, but obviously not good for a permanent solution.

Tony

Iain Westland

12-14-2008, 09:54 AM

count down to reset, with a nice reset now button next to it to pre empt the action and reset the computer

Naturally Digital

12-14-2008, 11:57 AM

Well... its obvious now that this IS a wrap-around problem... and adding bits will not solve the eventual fact that those variables will still wrap... only with a longer time interval... and it's also obvious that will never satisfy those who decided that this was a problem in the first place...

So... I am off to find the culprits... buried deep within the engine...Bob, I just want to say thank you for digging into this. Whether it's important or not, I appreciate your efforts and I appreciate your desire to make it as good as it can be. Thanks!

Bob L

12-14-2008, 02:35 PM

Tony... nice idea... but no go... resetting the engine by an internal stop/start will still glitch audio... and I'd like to fix it without having to do that.

Bob L

Jesse

12-14-2008, 10:15 PM

Bob, if you put it for 27 years, i can live with that, i doubt a computer will last that long as technology is evolving these days, not to mention that maybe SAC will run on CPU's with 80 cores or more by then, maybe assigning one core per channel :)

This is interesting because SAC not only it's redefining the way we do live mixing, it's also a new frontier in the programming side too, maybe discovering new ways of writing the code, "The Lentini Code" like Ira said :D

I think the engine it's the best and more stable of the market, but with this issue fixed nothing will stop it, unless something happens to my rig like an earthquake or tsunami. All right, i exagerated but you get the idea.

Thanks a lot for tracking this down Bob, your support is unique. :cool:

How can i help?

Jesse

Bob L

12-22-2008, 10:34 PM

Ok... my last test is still running at 29 1/2 hours... looks like this little invisible bug has been squashed. :)

Hopefully... anyway... I'm leaving it run... I'll see what happens.

Bob L

Tony Marone

12-22-2008, 11:38 PM

Bob,

COOL! So what was the problem??:) Now my customer on the first week of Feb won't have to be instructed what to do when the engine locks up!:o That would be quite embarrassing for me. I beez hopin' you got it bro!:cool:

Bob L

12-23-2008, 01:10 AM

It actually was a division instruction that caused the grief... numerator grew to be too big for the division limits... I had to re-design the way that math was handled.

I will still probably come up with a different solution down the road, but for now... at least the 27 hour limit has been lifted.

Bob L

Naturally Digital

12-23-2008, 11:38 AM

Tony Marone

12-28-2008, 10:57 AM

Hi Bob,

So what is the story? How long did you run your test for? Did it lock up again or is pretty much fixed? When might we get our hands on the fixed version to test for ourselves? Just full of questions, aren't I?:D I understand if you are pissed at me:o, fixing assembly language code is maddening. I really do appreciate your effort and what goes into it.

Tony