[nycbug-talk] File Backed Disks- Speed Issues

[Okan Demirmen]

On Thu 2006.09.28 at 14:15 -0400, Isaac Levy wrote:
> Hey All,
> 
> So I'm sad, and wanted to solicit some help to perhaps cheer me up.
> 
> SITUATION:
> I've been using File-Backed disks via mdconfig(8) with FreeBSD jail 
> (8) for a very long time.
> (I tend to call them 'disk images', Apple vocabulary.)
> 
> Creating a New File-Backed Disk with mdconfig:
> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/disks- 
> virtual.html#AEN24952
> -or-
> http://tinyurl.com/gqjjv
> 
> Their usefulness, in my enviornment, is that they are far more  
> flexible to work with in constraining disk size for jails across  
> machines than any other strategy, (namely, hard disk partitions).
> 
> --
> My sadness, and my query has to do with speed- or the painful lack of  
> it.
> Running a disk image is notably slower than the raw disk, (in my  
> case, fairly snappy RAID5/SATA), but 'usable'.
> Based on totally unscientific benchmarking, when I've got more than  
> twenty disk images on a given machine, (all running jails), disk I/O  
> becomes far slower than is usable- and each new disk takes the speed  
> down at what seems to be an order of significant magnitude.
> 
> What I'm saying is, they don't scale.
> 
> During the FreeBSD 4.x days, (wow what an era), file-backed disks  
> were snappy, with little overhead, no? (vn(4) and vnconfig(8),  
> respectively)

I can't comment on how md(4) is performing, but I did notice you said
RAID5/SATA. Have you created a case to eliminate RAID5/SATA? I believe
SATA is really mostly for just raw storage and not to be used to
anything that requires a reasonable amount I/O. I've run into situations
where we ended up tossing cabinets of SATA into just storage and online
backup, and replacing with SCSI; no more I/O issues. In another, moved
VMWare clusters from SATA to SCSI; again removing performace issues due
to I/O.

Note that I only have cabinet-size arrays of SATA experience, and not
4-5 SATA disks, so results may differ; as well as with application.

I think you might have to do more than "unscientific benchmarking." I'm
afraid that may help you narrow down the issue, but I do realize that'll
take some time...

Cheers,
Okan