[nycbug-talk] Memory sizing

[Francisco Reyes]

Pete Wright writes:
> the best i'd look for info on memory utilization in freebsd would be the
> McKusick/Neville-Neil design and implementation book.

Althought that is a good long term plan I would settle for a good 
explanation of top's memory display. :-)
I have found some, but not to the level I need.
In short I am looking to answer "does this machine has enough memory for the 
amount of programs we are running it it.. and to leave a decent amount of 
memory for the OS to cache files".

> text in general, and should help you get a handle on how FreeBSD in
> specific is going to utilize memory.

That is great to know.. but ultimately it comes down to.. does the machine 
has too much running on it.

The first thing I look at is swap. If the machine is using tens of megabytes 
from swap and you see the swap getting used frequently in top.. that is a 
clear sign that machine has too many processes for the amount of RAM.  

> what is the profile on queries you are going to running against the
> DB?

I am looking to size two machines. One DB, one mail server.
For the DB a large percent of the load will be writes. The app that will 
connect and the load programs are currently getting done and I don't have 
numbers yet.. However from the existing system (using mysql) that the new 
system will replace I see probably 90% + are writes with mostly relatively 
simple (but large) queries. Queries like "show me all the records that 
matchi this particular criteria and join a couple tables to expand codes".

>are we looking at lot's of small queries,periodic quries on large
> data set's or something in between.

Lots of inserts.
Unfrequent queries, but when they are done they will return very frequently 
large number of recores. As far as I can tell it will always be index 
access.

> large postgres server that is going to be doing data warehousing, we are
> thinking that we should be fine with 4gig's or RAM as we will not have
> many concurrent connections.

I am a bit undecided between going SATA or SCSI.
To be more precise..
SATA
6 400GB disks on RAID 10
2 hot spares
Net usable: 1.2TB raw (before filesystem)
8GB RAM
7.2K rpm disks

SCSI
7 150GB disks on RAID5
1 hot spare
Net usable: 900GB raw (before filesystem)
4 GB RAM
10K rpm disks

There is also the 10K rpm SATA drives, but I don't think RAID5 with SATA may 
be good enough. 

>From what I am reading so far after 6 disks the performance between RAID10 
and RAID5 is simmilar.

> if you are going to have many people hitting this thing, then yea i'd
> give it as much memory as practical and the fastest storage i can
> afford.

Found a link on postgresql performance and they actually claim that disk 
subsystem, them memory, then CPU is what matters.. Obviously as long as 
there is a minimal/acceptable amount of RAM to begin with.
 
> as an aside, i have another postgres DB that we are running out of a
> memory FS.

Interesting. How big is the DB?

Can't wait to the days when solid state drives become cheaper. :-)