Eric,
I have added some more comments below.
Regards,
Darren
Eric Boehm wrote:
> >>>>> "Darren" == Darren Keverne <darren.keverne@tiuk.ti.com> writes:
> >>>>> "Roy" == Roy Chapman <Roy.Chapman@de.bosch.com> writes:
>
> I am going to chime in with our own recommendations. We are in the process of
> trying to come up with recommendations corporate-wide.
>
> Darren> In all my configuration, we keep the views on the same machine as
> Darren> the VOBs to make view to VOB communications as fast as possible,
> Darren> and use remote storage of all the VOB pools and the view .s
> Darren> wherever possible.
>
> Roy> Kind of goes against all Rational recommendations. They strongly
> Roy> recommend that views and vob databases run on separate systems.
>
> Darren> I should have said at the top of the message what sort of things
> Darren> we do with ClearCase... We don't do a great deal of code
> Darren> development, (in the normal sense of the words), we do a lot of
> Darren> configuration management for environments to run tools in, and
> Darren> don't do a great deal of DO work. This means that our views can
> Darren> get very large with view private data (some are over 20GB) and our
> Darren> VOBs can also get large with checked in data (some over 10GB).
>
> I can see your point, but I believe that the majority of ClearCase users are
> probably doing code development. I know that we are. Quite often the code base
> is very large.
>
> Roy> that views and vob databases run on separate systems. For improved
> Roy> performance they recommend that the cleartext pools (c) are located
> Roy> on the view servers and, although not supported but recommended by
> Roy> their top performance guru Harry Abadi, also relocate the source
> Roy> pools (s) to the view server as well.
>
> This is rather difficult if you have more than one view server. We have 8 view
> servers.
>
> Darren> We have about 50 VOBs and nearly 200 views on one server, and it
> Darren> has around 100GB of total CC storage. I don't want a Sun server
> Darren> (no matter how good it is) doing the work of our file servers.
> Darren> The file servers can give the data to the clients, once they have
> Darren> found where it is from the server, much faster. This means the
> Darren> server just works out where it is.
>
> I have mixed opinions on the remote storage pools. I think the Suns do fine as
> file servers. I'm not sure that I want the complexity of trying to do
> administration/backups with remote storage pools.
>
The benefits we get from having powerful file servers outweighs the trouble of
backups. We use local MultiSite backups to make our life much easier... Most of
the remote data is in the views...
>
> I am willing (even eager) to be convinced otherwise. I know that some sites
> swear by using Auspex or Network Appliance for storage pools.
>
> Darren> The way things seem to work (we snooped the albd when we were
> Darren> trying to work out the best method) is that the views need to talk
> Darren> to the VOBs to find things, and then give them to the users, and
> Darren> with the view servers far away on the network (even next on the
> Darren> switch is not close enough) you get quite a performance hit.
>
> I'd be interested in what you did and how you arrived at your conclusion.
We spent quite some time monitoring the traffic around the network. The etherman
and packetman tools (from Curtin University) are very good at showing what is
going around the network, and the amount of traffic between our then two
VOB servers and one view server indicated that they would be better off being in
the same box.
> I
> already knew that the views want to talk to the VOBs but I am surprised that
> "next on the switch is not close enough".
>
> What kind of network do you have on your VOB server. 10BT? 100BT? ATM? Gigabit?
>
We have 7 100MB full-duplex ports coming out of the server. We have 2 100MB
switched networks and 5 100MB shared/10MB switched networks. Our server farm is
all on 100MB of some description...
>
> Darren> We do all of our work from distributed machines in a server farm,
> Darren> so we can't use the model of one build machine, with its views
> Darren> locally. This might be nice, but we have significantly more than
> Darren> one build machine, and we want to use the same view on all of
> Darren> them.
>
> I can see that an fast NFS server (i.e., Auspex, Network Appliance) might be
> useful here.
>
> We have a build farm of 38 E250s (2 CPU, 2GB RAM). We aren't using distributed
> builds right now -- because of problems with the Makefiles, but that's another
> discussion. So we use them for fast compile engines with local views.
What sort of makefile problems do you have???
>
>
> Darren> much more over this, consider getting more disks. I also strip
> Darren> all the disks together in the servers, to improve the speed. I
> Darren> don't bother with RAID, that's what backups are for. I would
> Darren> rather have access to the disks twice as fast and be up for 95% of
> Darren> the time, than have slower access and be up 98% of the time...
>
> I have to more strongly disagree here. I don't know if you mean RAID 1
> (mirroring) or RAID 5 (parity disk). I would recommend against RAID 5 but I
> think that no mirroring is asking for trouble.
>
> Our environment needs 99.9% availability. We do backups (obviously) but I
> don't ever want to use them. The effort involved in restoring from backups is
> too costly, any way you want to measure it!
>
We too would like 99.9% availability, and we more or less get that. All we do is
to strip the internal disks on the server to improve write performance. We
haven't had a need to do anything more with them. The way I have the backups
setup, we can perform a complete server rebuild fairly quickly... We are going to
time this some time soon, but I should imagine we could get a new server up and
running in a few hours...
>
> First, I'll start with our current environment.
>
> 2 E3500, 3 CPU, 6 GB RAM, 2 Sbus IO boards, 2 ATM interfaces per server.
> 2 A5000 FC-AL disk arrays, 14 x 18GB disks each array. Both hosts are dual
> connected to each array. 2 100GB volumes are mirrored between arrays. All
> connections to FC-AL disks are spread across two IO boards. The servers are
> directly connected via ATM (155 Mbps) to an ATM switch.
>
> We serve 150+ VOBS with 100GB of data to 350-400 desktops. We also MultiSite
> to 3 other locations.
>
We also run MultiSite, for backups to a local server, and to about 20 remote
servers. Not all having the same data, but they all talk to each other in the
end. This number is growing continuously...
>
> We will be expanding this with 4 more E3500s (partly to deal with the lock
> manager problem).
>
> Build farm is 38 E250s, 2 CPU, 1 or 2 GB RAM, 6 x 9GB or 6 x 18GB disks,
> mirrored, view space is striped across 2 disks. Connected via 100BT to
> Ethernet switch on ATM backbone.
>
> Darren> 1. For a small group, less than 10 designers and less than 10
> Darren> VOBs, and a few views per person, I normally recommend an E450,
> Darren> with 1GB of RAM, 2 CPUs and at least 10GB of local disk. This
> Darren> gives enough room to grow, without being too expensive to get
> Darren> started.
>
> I think you could get by with an E250 here.
>
> 1. 10-50 users, E250, 2 CPUs, 2GB RAM, 6 x 18 GB disks, mirrored (2 for OS, 4
> for data space) gives you about 30GB of disk.
>
> 2. 40-75 users, E450, 2 CPUs, 2GB RAM, 20 x 18 internal disks (mirrored). This
> would give you about 160GB internal disk
>
> Keep in mind, I am thinking about our environment where there are lots of
> DOs.
>
> 3. 40-100 users E450, 2 CPUs, 2GB RAM, 1 A5100 array (14 x 36 GB).
>
> 4. 75-150 users, E3500, 2 CPUs, 4 GB RAM, 1 A5200 array (22 x 18 GB).
>
> These configurations are meant to be scalable but not highly available.
>
> For highly available configurations
>
> 5. 80-200 users, 2 E450, (2 CPUs, 2GB RAM each server), dual-connected to 2
> A5x00 arrays.
>
> 6. 150-300 users, 2 E3500 (2 CPUs, 4 GB RAM, 2 IO boards each server),
> dual-connected to 2 A5x00 arrays.
>
> Configurations 3 and 4 are meant to be half of 5 and 6 respectively. If you
> start with 3 or 4, it is easy to grow to 5 or 6. The highly available
> configurations strive to reduce the number of single points of failure. At
> this time, it doesn't look like SunCluster will work with ClearCase but we
> continue to investigate it. The reason is that SunCluster has a requirement
> that cluster nodes cannot be NFS clients.
>
So how does CC cope with multiple servers for the same data, or are you serving
different data from the two servers, just using the same disk packs???
>
> Darren> 2. For a group of less than 40 designers and less than 100 VOBs,
> Darren> and a few views per person, I normally recommend an E3500, with
> Darren> 3-4GB or RAM, at least 4 CPUs and 20GB of local disk. This gives
> Darren> a lot more power and performance, with the FCAL disks and more
> Darren> memory. Memory is the most important thing when you start to get
> Darren> this big, I normally try to get as much physical memory as the
> Darren> used space on the local disk, so the whole database can be
> Darren> buffered in memory, and you don't have to go anywhere near the
> Darren> disk... This gets expensive.
>
> Unfortunately, in our case, it isn't possible to get the whole database into
> RAM. We would probably need on the order of tens of GB of RAM.
I am talking about the VOB db, not the whole VOB with the source. If you db is
that big, are you running 64bit 4.0 already???
>
>
> Darren> 3. For more than 40 people, or more than 100 VOBs, I normally
> Darren> recommend an E3500 fully loaded, that is with 8GB RAM and 8CPUs.
> Darren> The local disks need to be based on how big your VOBs are, you
> Darren> want at least twice as much local disk as the total VOB DB size,
> Darren> more is better and try to keep the used space below 40% of the
> Darren> total space.
>
> I think this might be a problem. If you have lots of VOBs on one machine, your
> biggest bottleneck is likely to be the lock manager. The lock manager (and
> mvfs) are not multi-threaded. I don't think that 8 CPUs would be fully
> utilized. Also, you are only allowing for one IO board. I think that you are
> sacrificing IO performance for CPU and memory.
>
As far as I know, you don't need mvfs on the server, so that does no count, and
for the lock manager, this seems to serialize calls to all the VOBs, but does not
stop work in one VOB because something is still going on in a different VOB. This
just stops two accesses into the same VOB at the same time...
>
> Darren> As you get bigger than this, you need to start clustering your
> Darren> servers. If a process on one talks to a process on the other,
> Darren> they need to be as close as possible to each other on the network.
> Darren> I normally put as many network interfaces in the servers as there
> Darren> are distinct subnets to talk to. In my UK server this means 7
> Darren> different ports.
>
> I agree that you should cluster your servers but I think you should start
> earlier.
>
> Most of our clients are on the same subnet as the server. If they aren't,
> then you will need multiple interfaces (as you say).
>
> Darren> If you have NT in the mix, I normally provide direct access to the
> Darren> server via something like Samba, so the clients can get to the
> Darren> data, and normally serve the data from the file server through the
> Darren> CC server, but this puts a strain on the server, so get more
> Darren> memory if you need to do this...
>
> Again, I agree. If the storage pools are on the CC server, then you avoid some
> of the strain.
>
> Keep in mind that these configurations are envolving -- this is our starting
> point.
>
> --
> Eric M. Boehm boehm@nortelnetworks.com
-- ====================================================================== | Darren L Keverne _ Email : dkeverne@ti.com | | CM Methodology _| '-. Mobile : dkeverne@bulletinmail.com | | ASIC Product Development \, _} MS : TIL/10 | | Texas Instruments UK \( Phone : +44 (1604) 663474 | | Northampton England Fax : +44 (1604) 663456 | ======================================================================- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
This archive was generated by hypermail 2b29 : Sun May 06 2001 - 00:23:16 EDT