ZFS configuration advice, post Veritas

[Forrest]

I need some advice on handling our new zpool.

We first received the x4540 thumper which came preconfigured with
raidz, which worked fine. Then, someone in our dept decided we needed
Veritas, which ended up not working out (I hate that product
anyway). Now, we're backing out Veritas, flattening the volumes to
free up disk space for creation of a zpool, where the data will be re-
migrated.

I read a couple of articles on "best practices" out there. It seems
that mirroring is the most recommended solution. The server has 30+
terabytes of data, but we are quickly eating it up with the media we
store, which is mostly video files and their associated data. These
volumes are in turn made available internally via NFS for different
purposes.

We've managed to free up these disks for the initial zpool, which I'll
then add others to (presuming mirroring):

c1t2d0s2
c1t3d0s2
c5t2d0s2
c5t4d0s2
c6t3d0s2
c6t4d0s2
c6t6d0s2
c6t7d0s2

I read about building your storage using dynamically striped RAID-Z
groups of (Y / X) devices. Sounds a little complicated to me.

So before I go in and make a config that I can't easily back out of
(grin), I thought someone out there might have some advice/tips about
the zpool config I build.

Another side issue is that we've been using rsync to replicate our
nightly data (runs hourly or so). We're going to back this out and
start sending ZFS snapshots to our remote thumper (similar system),
but I've read that there are problems maintaining metadata like NFS
handles and such, which must be in order on the remote system (it's a
failover). Anyone know the currents status of that issue?


Thanks in advance!

[Ian Collins]

Post by Forrest
I need some advice on handling our new zpool.
We first received the x4540 thumper which came preconfigured with
raidz, which worked fine. Then, someone in our dept decided we needed
Veritas, which ended up not working out (I hate that product
anyway). Now, we're backing out Veritas, flattening the volumes to
free up disk space for creation of a zpool, where the data will be re-
migrated.
I read a couple of articles on "best practices" out there. It seems
that mirroring is the most recommended solution. The server has 30+
terabytes of data, but we are quickly eating it up with the media we
store, which is mostly video files and their associated data. These
volumes are in turn made available internally via NFS for different
purposes.
We've managed to free up these disks for the initial zpool, which I'll
c1t2d0s2
c1t3d0s2
c5t2d0s2
c5t4d0s2
c6t3d0s2
c6t4d0s2
c6t6d0s2
c6t7d0s2
I read about building your storage using dynamically striped RAID-Z
groups of (Y / X) devices. Sounds a little complicated to me.

How you build your pools depends on may things, some important ones are:

how much redundancy do you require?
How long are you willing to risk the pool not having any redundancy
during a drive replacement?
What performance characteristics are you after?

I typically use stripes of mirrors for data that will be randomly read a
written (general file servers) and stripes of raidz2 (6+2) for pools
that will be used for data read and written sequentially. This sounds
like the type of data you will be storing.

Post by Forrest
Another side issue is that we've been using rsync to replicate our
nightly data (runs hourly or so). We're going to back this out and
start sending ZFS snapshots to our remote thumper (similar system),
but I've read that there are problems maintaining metadata like NFS
handles and such, which must be in order on the remote system (it's a
failover). Anyone know the currents status of that issue?

I've never heard of this issue, where do you read about it? zfs
send/receive is way faster than rsync for large pools.

[Forrest]

Post by Ian Collins

Post by Forrest
I need some advice on handling our new zpool.
We first received the x4540 thumper which came preconfigured with
raidz, which worked fine.  Then, someone in our dept decided we needed
Veritas, which ended up not working out (I hate that product
anyway).   Now, we're backing out Veritas, flattening the volumes to
free up disk space for creation of a zpool, where the data will be re-
migrated.
I read a couple of articles on "best practices" out there.  It seems
that mirroring is the most recommended solution.   The server has 30+
terabytes of data, but we are quickly eating it up with the media we
store, which is mostly video files and their associated data.   These
volumes are in turn made available internally via NFS for different
purposes.
We've managed to free up these disks for the initial zpool, which I'll
c1t2d0s2
c1t3d0s2
c5t2d0s2
c5t4d0s2
c6t3d0s2
c6t4d0s2
c6t6d0s2
c6t7d0s2
I read about building your storage using dynamically  striped RAID-Z
groups of (Y / X) devices.  Sounds a little complicated to me.

how much redundancy do you require?
How long are you willing to risk the pool not having any redundancy
during a drive replacement?
What performance characteristics are you after?
I typically use stripes of mirrors for data that will be randomly read a
written (general file servers) and stripes of raidz2 (6+2) for pools
that will be used for data read and written sequentially.  This sounds
like the type of data you will be storing.
[ ... ]
I've never heard of this issue, where do you read about it?  zfs
send/receive is way faster than rsync for large pools.
--
Ian Collins

We have a mix of data that live on the NAS (thumper). Since this is
the core of our operation, redundancy is fairly important. The stock
raidz configuration worked fine for our needs. The problem I'm faced
with now is that we're gradually backing out of Veritas, which means I
have a limited number of free disks available for the zpool, and must
add the freed-up disks once I copy data over and the subsequent
Veritas volume is destroyed. This presents a problem with raidz,
since you can't append. I'm not sure how many drives I'll have
free during the procedure, but I can't see us having multiple raidz's
striped. Some of them will not be uniform.

So it really looks like our only choice is mirror at this point.

[Ian Collins]

Post by Forrest

Post by Ian Collins
how much redundancy do you require?
How long are you willing to risk the pool not having any redundancy
during a drive replacement?
What performance characteristics are you after?
I typically use stripes of mirrors for data that will be randomly read a
written (general file servers) and stripes of raidz2 (6+2) for pools
that will be used for data read and written sequentially. This sounds
like the type of data you will be storing.
[ ... ]
I've never heard of this issue, where do you read about it? zfs
send/receive is way faster than rsync for large pools.

We have a mix of data that live on the NAS (thumper). Since this is
the core of our operation, redundancy is fairly important. The stock
raidz configuration worked fine for our needs. The problem I'm faced
with now is that we're gradually backing out of Veritas, which means I
have a limited number of free disks available for the zpool, and must
add the freed-up disks once I copy data over and the subsequent
Veritas volume is destroyed. This presents a problem with raidz,
since you can't append. I'm not sure how many drives I'll have
free during the procedure, but I can't see us having multiple raidz's
striped. Some of them will not be uniform.
So it really looks like our only choice is mirror at this point.

Mirrors are generally a safe bet. If you have anything other than tiny
drives stay well clear of raidz, use raidz2 (or raidz3 for > 2TB
drives). If you want the extra space and can allocate drives 6 or 8 at
a time, a stripe of raidz2 should be fine.

[Forrest]

Post by Ian Collins
how much redundancy do you require?
How long are you willing to risk the pool not having any redundancy
during a drive replacement?
What performance characteristics are you after?
I typically use stripes of mirrors for data that will be randomly read a
written (general file servers) and stripes of raidz2 (6+2) for pools
that will be used for data read and written sequentially.  This sounds
like the type of data you will be storing.
[ ... ]
I've never heard of this issue, where do you read about it?  zfs
send/receive is way faster than rsync for large pools.

We have a mix of data that live on the NAS (thumper).   Since this is
the core of our operation, redundancy is fairly important.   The stock
raidz configuration worked fine for our needs.   The problem I'm faced
with now is that we're gradually backing out of Veritas, which means I
have a limited number of free disks available for the zpool, and must
add the freed-up disks once I copy data over and the subsequent
Veritas volume is destroyed.   This presents a problem with raidz,
since you can't append.     I'm not sure how many drives I'll have
free during the procedure, but I can't see us having multiple raidz's
striped.  Some of them will not be uniform.
So it really looks like our only choice is mirror at this point.

Mirrors are generally a safe bet.  If you have anything other than tiny
drives stay well clear of raidz, use raiz2 (or raidz3 for > 2TB
drives).  If you want the extra space and can allocate drives 6 or 8 at
a time, a stripe of raidz2 should be fine.
--
Ian Collins

There are 47 drives on this Thumper, of type:

ATA-HITACHIHUA7210S-AC5A cyl 37367 alt 2 hd 255 sec 205

I think these are 2T drives.

The first 2 are the root zpool.

Most of the data we store there are video files. But there are some
random I/O's of smaller files during production; so a good mix of
both.

The problem we have now is we have a limited number of disks that are
available (6 or 7) for a new zpool. From there, we'll copy data over
from the older Veritas volume, then destory it and detach the
associated disk(s). Cumbersome. I'd like to keep the configuration
simple.

I'm concerned that a mirror could fail in the event of >2 drive
failures.


Thanks!