Session over network

[Cary B. Cornett]

I just picked up a surplus server that implements a RAID 5 array. I was told yesterday that pulling files over a network used less processor time/power than pulling files from a local drive. Is this true? Can SS work with all session files on a network server instead of on the local machine? Are there any special considerations for doing this, such as different buffer settings?

Also, is anyone here using SS in this way, and if so what is your experience with it?

[Craig Allen]

It's probably true, but the bottleneck will likely be the network itself. You didn't mention what speed it was running at but that will make a difference. If you hook it up, you could always try Bob's HD speed test to see how it works.

[DamonD]

Just a little tidbit on RAID... RAID5 is the worst for any drive perfomance but best for redundancy, if you can reconfigure them for RAID 0 (striping across multiple disks), RAID 1 (Mirroring) or RAID 0+1 or 1+0 (Mirror and stripe), you'll be better off.

So the RAID5 will most likely hinder things a bit, more for writes than reads though.

Good luck
Damon

[Bob L]

I have pulled many large sessions live over my network, even when it was 100 baseT... now with a GigaBit network... its even easier.

You will just have to try it and see.

Bob L

[DominicPerry]

I see no reason why reading files from an HD controller on the PCI bus should take any more CPU than reading them from a Network card on the PCI bus. The only difference is how fast can the two deliver the data? This has changed a lot over the past 10 years. HD controllers do most of the work to shift the data around from the HD to memory, and likewise network controllers have enough onboard power to run the ip stack and shift data without much intervention from the CPU. The easiest way to tell is to copy a huge file from the internal hd to an external hd and then do the same from network drive to external hd. Watch the CPU, time the transfer. Then do the same with lots of little files.

The old style PCI bus is about 132MB/s which is 1056Mb/s. Gbit ethernet is 1000Mb/s. But ethernet has larger overheads, so I imagine that it would be a little slower under those circumstances. Also bear in mind that you can control the PCI latency of devices connected to the PCI bus. And my experience is also that a flaky network drive will distract the OS a lot, whereas a local HD will not, unless it spins down.
Of course, PCI-E is much faster, and I don't know if there are any PCI-E based HD controllers, but the limiting factor is likely to be, is the drive able to provide the data?

Dominic

[UpTilDawn]

... And my experience is also that a flaky network drive will distract the OS a lot, whereas a local HD will not, unless it spins down.


I can attest to the flaky network drive (or the network or card as well) kind of issues. I've experienced copies of tracks with corrupted and missing data before because of it. Shortly after the third or fourth bad transfer my gigabit switch died. I haven't had an issue since replacing it.

DanT

[Cary B. Cornett]

I'm still using a 100 megabit network, which so far has been quite fast enough for anything I have needed. The main purpose of the server is to provide the best possible storage reliability, so the redundancy of RAID 5 trumps speed in this case. I haven't been expecting to do serious session work over the network, but having the capability might be pretty cool.

The whole thing that drove the idea of a server was the realization that I have way too much work that could be lost by the failure of a single drive. A friend suggested RAID 5 as the best protection against loss, since the failure of any single drive in the array does not lose any data, so long as that drive is replaced and the system is allowed to "rebuild" its contents. In fact, using RAID 5 actually lets me get away with using used/surplus drives for the array, allowing me to have more storage for less money. After all, even with old drives, they generally fail one at a time.

So far this is more an experiment, as I an in the process of setting up the server. I only just now finished the initial OS install... lots to do yet. Meanwhile, the answers I am getting here are helping me in my choices. :cool:

[DominicPerry]

I don't know whether this can already be done using SAW's TCP/IP link, but I'd really like the option of a second, active backup path, so that you can write to local and remote drive (either external HD or network path) at the same time. Lots of location recorders do this (local HD + DVD or Compact Flash). For 'paranoia', that would cover it. There is, of course an overhead on the machine, a possible delay in SAW, and of course the overhead of Bob coding it, I don't know what the demand would be. I personally would use it but it's not vital for what I do.
An alternative is a 'diskcopy' utility which monitors a drive and copies it to a remote drive. These utilities already exist. I think RoboCopy will do this.

Dominic

[Bob L]

I simply use the Save Copy As command every so often as I work on a session... easy enough to do and the path stays put so there is no further typing needed once you do the first one.

I prefer to keep the background apps to a minimum.

Bob L

[Dave Labrecque]

I don't know whether this can already be done using SAW's TCP/IP link, but I'd really like the option of a second, active backup path, so that you can write to local and remote drive (either external HD or network path) at the same time. Lots of location recorders do this (local HD + DVD or Compact Flash). For 'paranoia', that would cover it. There is, of course an overhead on the machine, a possible delay in SAW, and of course the overhead of Bob coding it, I don't know what the demand would be. I personally would use it but it's not vital for what I do.
An alternative is a 'diskcopy' utility which monitors a drive and copies it to a remote drive. These utilities already exist. I think RoboCopy will do this.

Dominic

Dominic,

Do you mean more or less real time copying? I do a nightly backup of my system's changed files to a separate HD, which seems to have worked for me so far... (fingers crossed) Obviously, a crash in the middle of the day of main drive can hose me vis a vis any work done that day.

[DominicPerry]

Doing what Bob says is probably the easist and most sensible approach, I understand about keeping background processes to a minimum.
But yes, I was talking about realtime replication/duplication. As I said, location recorders like the HHB Portadrive and Zaxcom Deva record to internal HD and DVD at the same time. This is great cos you can keep a copy and send the other off as rushes as soon as the shoot/session finishes or keep one as a backup. No copying time needed, and if one of the media fails during recording, the other keeps going, so no need to re-run a whole day's expensive shoot. And an avoidance of embarassment/firing. Nice.

I've checked and Robocopy won't do this because it can't copy open files. I'll see if I can find something which will.

Dominic

[UpTilDawn]

But yes, I was talking about realtime replication/duplication. As I said, location recorders like the HHB Portadrive and Zaxcom Deva record to internal HD and DVD at the same time. This is great cos you can keep a copy and send the other off as rushes as soon as the shoot/session finishes or keep one as a backup. No copying time needed, and if one of the media fails during recording, the other keeps going, so no need to re-run a whole day's expensive shoot. And an avoidance of embarassment/firing. Nice...

Great device unless the machine itself breaks down, then you lose connection to the main AND the backup at the same time. Only way around that is to use two separate devices with a splitter from the source.

DanT

[DominicPerry]

Great device unless the machine itself breaks down, then you lose connection to the main AND the backup at the same time. Only way around that is to use two separate devices with a splitter from the source.

DanT

Agreed, but these are low track count (6 - 10) devices at huge cost ($10,000 - $20,000), so they are built to 'not fail'. I agree that a laptop with Windows isn't so reliable. But SAW probably is, cetainly the closest I've ever come to a piece of bullet-proof software, except perhaps 'Notepad'.

Dominic

[bcorkery]

except perhaps 'Notepad'. :D

[Naturally Digital]

An alternative is a 'diskcopy' utility which monitors a drive and copies it to a remote drive. These utilities already exist.There's also Mirrorfolder http://www.techsoftpl.com/backup/index.php

I've checked and Robocopy won't do this because it can't copy open files. I'll see if I can find something which will.It appears this mode exists in Mirrorfolder: http://www.techsoftpl.com/backup/faq_view.php?id=4

[Dave Labrecque]

There's also Mirrorfolder http://www.techsoftpl.com/backup/index.php
It appears this mode exists in Mirrorfolder: http://www.techsoftpl.com/backup/faq_view.php?id=4



Just a thought guys... I considered a real-time mirror type approach a while back, but decided there was no way I wanted that kind of activity cutting into my performance during tracking or editing. That's why I do a nightly backup.

Is this thinking out-dated? :confused:

[DominicPerry]

Since my last post I tried about 6 packages which claim to do this. None of them did except mirrorfolder. However, although it continued to write to the primary target when the secondary was full, it subsequently hung my machine and the OS needed a bit of work to recover it. I doubt if it would continue writing to the 'mirror' if the primary filled up or failed. But the test sort of proved that a hardware (RAID) solution would probably be the best. The CPU overhead was negligible though.

Dominic

[studio-c]

But SAW probably is, cetainly the closest I've ever come to a piece of bullet-proof software, except perhaps 'Notepad'.

Dominic

Luckily, I can type 1's and 0's at 44,100 characters per second. :D