[nycbug-talk] File Backed Disks- Speed Issues

[Isaac Levy]

Hi All,

On Sep 28, 2006, at 4:08 PM, Okan Demirmen wrote:

> On Thu 2006.09.28 at 14:15 -0400, Isaac Levy wrote:
>> Hey All,
>>
>> So I'm sad, and wanted to solicit some help to perhaps cheer me up.
>>
>> SITUATION:
>> I've been using File-Backed disks via mdconfig(8) with FreeBSD jail
>> (8) for a very long time.
>> (I tend to call them 'disk images', Apple vocabulary.)
>>
>> Creating a New File-Backed Disk with mdconfig:
>> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/disks-
>> virtual.html#AEN24952
>> -or-
>> http://tinyurl.com/gqjjv
>>
>> Their usefulness, in my enviornment, is that they are far more
>> flexible to work with in constraining disk size for jails across
>> machines than any other strategy, (namely, hard disk partitions).
>>
>> --
>> My sadness, and my query has to do with speed- or the painful lack of
>> it.
>> Running a disk image is notably slower than the raw disk, (in my
>> case, fairly snappy RAID5/SATA), but 'usable'.
>> Based on totally unscientific benchmarking, when I've got more than
>> twenty disk images on a given machine, (all running jails), disk I/O
>> becomes far slower than is usable- and each new disk takes the speed
>> down at what seems to be an order of significant magnitude.
>>
>> What I'm saying is, they don't scale.
>>
>> During the FreeBSD 4.x days, (wow what an era), file-backed disks
>> were snappy, with little overhead, no? (vn(4) and vnconfig(8),
>> respectively)
>
> I can't comment on how md(4) is performing, but I did notice you said
> RAID5/SATA. Have you created a case to eliminate RAID5/SATA? I believe
> SATA is really mostly for just raw storage and not to be used to
> anything that requires a reasonable amount I/O. I've run into  
> situations
> where we ended up tossing cabinets of SATA into just storage and  
> online
> backup, and replacing with SCSI; no more I/O issues. In another, moved
> VMWare clusters from SATA to SCSI; again removing performace issues  
> due
> to I/O.

Well, erm- I've found the sata systems to be every bit as snappy at  
this scale- (4 drives per 1u box).

>
> Note that I only have cabinet-size arrays of SATA experience, and not
> 4-5 SATA disks, so results may differ; as well as with application.

Gotcha.  At my scale, honestly, I've found no real i/o problems with  
SATA over SCSI.  Technically, I understand SCSI is faster, but the  
margins are so low at this scale it becomes moot for me- especially  
over the price and ease of use.

>
> I think you might have to do more than "unscientific benchmarking."  
> I'm
> afraid that may help you narrow down the issue, but I do realize  
> that'll
> take some time...
>
> Cheers,
> Okan

I'm gonna' grok the source, and go back historically on the topic to  
see if I'm just dreaming about the past or some bs...  And should  
just move on.  This is beginning to smell like the kind of scale/ 
abstraction problem that doesn't go away in computing...

Rocket-
.ike