Learn Linux, 302 (Mixed environments)
Performance tuning
Measure and improve your Samba performance
Content series:
This content is part # of # in the series: Learn Linux, 302 (Mixed environments)
This content is part of the series:Learn Linux, 302 (Mixed environments)
Stay tuned for additional content in this series.
In this article, learn how to:
- Measure Samba performance
- Optimize Samba memory usage
- Improve file-transfer speeds in a Server Message Block/Common Internet File System environment
This article helps you prepare for Objective 315.3 in Topic 315 of the Linux Professional Institute's (LPI) Mixed Environment Specialty exam (302). The objective has a weight of 1.
Prerequisites
To get the most from the articles in this series, you should have an advanced knowledge of Linux and a working Linux system on which you can practice the commands covered in this article. You should also have a good understanding of TCP/IP networking.
Measuring Samba performance
Before you can improve something, you must be able to reliably measure it. Measure something, make changes, measure again, and compare results. In the case of Samba, you need to measure:
- Response time and throughput of a client under idle server conditions
- Response time and throughput of a client under a given server load
- Server characteristics under a given load
- Maximum server capacity in terms of clients or throughput
Measuring response times gives you an idea of what a typical client would expect under the test conditions. For this to work, your test cases should approximate actual client load. For example, seeing how fast you can repeatedly request the same file is not the same as copying a directory containing a mix of small and large files.
Measuring server characteristics under a given load gives you some idea of how much room you have to grow. If, under load, your server is struggling to keep up, then you know you don't have much capacity beyond the test load. The same techniques are used to measure a production server's load and extrapolate capacity.
Finally, the so-called "torture tests," where you throw all you can at a server and see where it breaks, provide interesting information but are not as useful as the other types of tests. If the goal is to prove that your server can handle a certain load, then this type of test will do. Such tests are often more helpful for measuring the lower-level characteristics, such as the maximum disk I/O capacity of a server.
Samba is a network server, so it is important to reasonably simulate the network being used. If your clients are 50 milliseconds away from the server, the effects of network latency will be more pronounced than with local clients. This information directs your tuning priorities accordingly.
Designing a test
You could buy a fairly expensive device that simulates client traffic and takes precise performance measurements. And if you're publishing benchmarks or developing the server hardware, such a device might be a good option. However, if you're interested in tuning your own server, and are faced with the usual constraints of time and money, you're probably not looking for an expensive tool that you'll have to learn how to use.
As an example, the first test assesses random read performance by downloading a large number of files with the Samba command-line client. The primary concern is, "how fast can this directory be downloaded?"
Like any well-behaved UNIX® utility, the
smbclient
tool can read its list of
instructions from the standard input. The following snippet shows a series
of commands for downloading the contents of a directory:
prompt recurse mget smbtest
The prompt
command suppresses the client from
asking you for confirmation of downloads, and
recurse
indicates that you want to descend into
directories when you download multiple files. Finally,
mget smbtest
instructs the client to start
downloading the smbtest directory. Fill that directory with a few hundred
megabytes of test files, and you have all you need to test performance.
To run the test, connect to your share with
smbclient
, and redirect standard input to your
file. Listing 1 shows how you do it.
Listing 1. Running the test
$ time smbclient '\\192.168.1.1\test' password < instructions Domain=[BOB] OS=[Unix] Server=[Samba 3.5.8-75.fc13] getting file \smbtest\file2 of size 524288000 as file2 (5323.2 kb/s) (average 5323.2 kb/s) getting file \smbtest\file1 of size 139460608 as file1 (5275.3 kb/s) (average 5313.0 kb/s) real 2m2.289s user 0m0.509s sys 0m4.580s
The command in Listing 1 starts with the time
command, which times how long the command specified in the rest of the
arguments takes to run. The command itself is
smbclient
, and the arguments to that are the
name of the share and the user's password. You can add the typical
arguments, such as -U
to pass an alternate user
name if your environment requires it. Finally,
< instructions
redirects the standard input
of smbclient
to the file called
instructions, which contains the instructions from the first
code snippet. The result of this command is a timed batch copy of several
files.
The result of the command is a list of the files transferred, along with an
average transfer rate per file. The output of
time
is added at the end, showing that copying
roughly 664MB of files took 2 minutes and 2.289 wall-clock seconds. This
is now the benchmark. If you make any changes and the test takes longer
than 2 minutes, 2 seconds, then you're making things worse with your
changes.
If you want to test just Samba parameters and negate the effects of local disk and caching, you can run the test several times and take the last measurement. Doing so ensures that the operating system caches as much as possible and minimizes disk access. As you test your changes, make sure you have a similar amount of free memory on your server; otherwise, the differences in the amount of cached data might alter the results of your experiment.
Viewing Samba's status
Looking at a server's CPU, memory, disk, and network information gives you
some good information about the health of the server itself but provides
no context for what the application is doing. Samba comes with a helpful
utility called smbstatus
that shows current
connections and file activity; it's also good for both performance tuning
and troubleshooting.
Listing 2 shows the
smbstatus
command in action.
Listing 2. The smbstatus command
$ smbstatus lp_load_ex: refreshing parameters Initialising global parameters params.c:pm_process() - Processing configuration file "/etc/samba/smb.conf" Processing section "[global]" Processing section "[homes]" Processing section "[printers]" Processing section "[extdrive]" Samba version 3.5.8-75.fc13 PID Username Group Machine ------------------------------------------------------------------- 17456 fred fred macbookpro-d0cd (::ffff:192.168.1.167) Service pid machine Connected at ------------------------------------------------------- fred 17456 macbookpro-d0cd Mon Jul 18 07:36:46 2011 extdrive 17456 macbookpro-d0cd Mon Jul 18 07:36:46 2011 Locked files: Pid Uid DenyMode Access R/W Oplock SharePath Name Time ---------------------------------------------------------------------------------- 17456 505 DENY_NONE 0x100081 RDONLY NONE /home/fred . 17456 505 DENY_NONE 0x100081 RDONLY NONE /home/fred Documents
Listing 2 shows the current activity of the Samba server. One user, fred, is connected, and he has two shares mounted (fred and extdrive). There are also two locked directories.
Network tuning
Samba is a daemon that primarily sends and receives packets over the network. Several parameters alter how the packets are sent and how Samba interacts with the underlying operating system; these parameters can have a drastic effect on performance.
The basic options
Many daemons provide the best possible level of service to everyone and do not discriminate among clients. If the service is overloaded, everyone gets the same bad service. Contrast this to a telephone network: if congestion occurs, people aren't allowed to make new calls, but existing calls continue as though nothing's wrong. If you artificially limit certain options in Samba, you can prevent getting to a resource starvation situation.
The max connections
parameter limits the number
of connections you can have to a particular Samba server. Each connection
takes memory and CPU resources, so it's possible to overload a server by
having too many connections. By default, unlimited connections are
allowed. If you need to limit a busy server, you can specify a hard limit
with max connections
.
The max smbd processes
parameter controls the
maximum number of processes that can be run. This parameter is similar to
max connections
but controls the number of
processes resulting from the connections.
The more you log with log level
, the more server
resources are spent writing logs to disk. Keeping this value at 1 or 2
limits the amount of logs written to disk and saves more resources for
serving clients.
Setting socket options
When an application asks the operating system to open a network connection, the application can also ask that the operating system treat the packets a certain way using socket options. Socket options can enable or disable network performance tweaks, set particular quality-of-service bits on the packets, or set kernel-level options on the socket.
The socket options
command can control the
type of service (TOS) bits on the packets. The TOS bits tell
routers along the way how the traffic should be treated. If the routers
are configured to respect these bits, the traffic can be handled in the
manner the application asked for. The
IPTOS_LOWDELAY
keyword is most appropriate for
low-delay networks such as LANs, while
IPTOS_THROUGHPUT
is for higher-latency WAN
links. Your network may be configured differently, so it's possible that
using the options might have the opposite effect.
The TCP_NODELAY
option disables the Nagle
algorithm (see Related topics for a link to more
information), which is more appropriate for chatty protocols, at a cost of
more packets being sent.
If you have firewalls or other devices that keep state in your network, you
may be interested in the SO_KEEPALIVE
option,
which turns on TCP keepalives. These periodic packets keep the connection
open and keep the state fresh inside firewalls. Otherwise, the firewall
will drop the packets, and it will take some time for the client to
realize it has to reconnect to the server.
Beyond tuning
Although playing with different settings and trying to come up with optimal configurations may be fun, some items outside the Samba configuration can really hamper performance. Any time the server is doing something other than reading data from disk and sending it to a client or receiving incoming network traffic and writing it to disk, you're making the operation take longer.
Ethernet errors
If a packet is lost in transmission, the kernel has to notice that the
packet has gone missing and request a retransmission. This process can
slow down a fast conversation, especially if the two sides are separated
by high latency. One common source of packet loss is mismatched settings
between the switch and the server. Either auto-negotiation fails to come
up with a correct match or one side is set statically and the other side
is still auto-negotiating. This discrepancy results in errors on both
sides. You can look for such errors with the
netstat
command:
# netstat -d -i 2 Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth1 1500 0 5258404 0 0 0 3024340 0 0 0 BMRU eth1 1500 0 5258409 0 0 0 3024341 0 0 0 BMRU eth1 1500 0 5258411 0 0 0 3024342 0 0 0 BMRU
This code shows the multipurpose netstat
command
looking for interface errors every 2 seconds with the
-d -i 2
parameters. Every 2 seconds, the status
of the available interfaces is shown. In the example above, the
RX-OK
and TX-OK
columns show that packets are flowing in and out. The errors, drops, and
overruns are all 0 in both directions, showing that there has been no
packet loss.
If you see errors, determine what speed/duplex is being used with
mii-tool
or
mii-diag
. Listing 3 shows
how to check your network settings.
Listing 3. Verifying network settings
# mii-tool eth1: negotiated 100baseTx-FD, link ok # mii-diag eth1 Basic registers of MII PHY #24: 3000 782d 0040 6177 05e1 41e1 0003 0000. The autonegotiated capability is 01e0. The autonegotiated media type is 100baseTx-FD. Basic mode control register 0x3000: Auto-negotiation enabled. You have link beat, and everything is working OK. Your link partner advertised 41e1: 100baseTx-FD 100baseTx 10baseT-FD 10baseT. End of basic transceiver information.
Listing 3 starts with the mii-tool
command to
give a short summary of the active interfaces. The results show that the
interface has negotiated at 100/Full. mii-diag
shows more detailed information and might be more suitable for sending to
your network team if you have to escalate the issue. The
mii-tool
command can be used to fix the speed
and duplex, though in practice it's better to have the link
auto-negotiated.
Maintaining TDB files
In "Learn Linux, 302 (Mixed environments): Trivial Database files," you learned about the trivial database files
(TDBs) that Samba uses to keep state. If something is wrong with these
databases, your server will have to perform more work, such as unnecessary
disk seeks, or fail to cache information from remote services.
Fortunately, you can check the TDB files for corruption and fix them if
there's a problem. You can delete the temporary TDB files after shutting
down Samba, and they'll be re-created on startup. For others, be sure to
perform a backup, and then verify the files with
tdbbackup -v
.
Look at the client
Sometimes, the client may be the cause of slow performance. The client might have duplex problems, it might have malware, or some other problem may be occurring. Using a consistent client for your testing can help you eliminate the server as a potential source of slowness.
Conclusion
This was the final article in the LPIC exam 302 series. Review your notes, and take the exam! Good luck!
Downloadable resources
Related topics
- The smb.conf man page has more examples and descriptions of the commands shown in this article.
- Dr. Gunther of Performance Dynamics has published books dealing with performance analysis and tuning. You might be familiar with his PDQ method of analysis from the LPIC 301 exam.
- Nagle's algorithm can have a profound effect on network speed.
- Review the entire LPI exam prep series on developerWorks to learn Linux fundamentals and prepare for systems administrator certification based on LPI exam objectives prior to April 2009.
- If you're looking for open-source performance testing tools, look at this list from opensourcetesting.org.
- Wireshark is the best packet analysis tool out there; use it to look at the packets your server is sending and receiving.
- The IOzone filesystem benchmark helps you figure out the speed of your storage.
- Download a 30-day trial of IBM Rational Performance Tester to help you identify bottlenecks in system performance.
- In the developerWorks Linux zone, find hundreds of how-to articles and tutorials, as well as downloads, discussion forums, and a wealth of other resources for Linux developers and administrators.
- Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.
- Follow developerWorks on Twitter, or subscribe to a feed of Linux tweets on developerWorks.