Learn Linux, 302 (Mixed environments): Trivial Database files

Samba's Trivial Database files and how to troubleshoot them

Samba uses Trivial Database files to store both persistent and temporary data as part of its job integrating file and print sharing between Linux and Windows. In preparation for the Linux Professional Institute Certification exam LPI-302, learn all about the Samba Trivial Database (TDB) format that Samba uses to store information, how to look inside TDB files, and how to back them up.

Share:

Sean A. Walberg, Senior Network Engineer

Photo of Sean WalbergSean Walberg is a network engineer and the author of two books on networking. He has worked in several industries, including health care and media.



08 March 2011

Also available in Russian Japanese Portuguese Spanish

About this series

This series of articles helps you learn Linux systems administration tasks. You can also use the material in these articles to prepare for the Linux Professional Institute Certification level 3 (LPIC-3) exams.

See our developerWorks road map for LPIC-3 for a description of and link to each article in this series. The road map is in progress and reflects the current objectives (November 2010) for the LPIC-3 exams. As each article is completed, it is added to the road map.

Overview

In this article, learn about these concepts:

  • Backing up Samba Trivial Database (TDB) files
  • Restoring TDB files
  • Identifying TDB file corruption
  • Editing/listing TDB file content

This article helps you prepare for Objective 310.3 in Topic 310 of the LPI's Mixed Environment specialty exam (302). The objective has a weight of 1.

Prerequisites

To get the most from the articles in this series, you should have an advanced knowledge of Linux and a working Linux system on which you can practice the commands covered in this article. In particular, this article assumes that you have a working knowledge of Linux command-line functions and at least a general understanding of the purpose of Samba as covered in "Learn Linux, 302 (Mixed environments): Concepts". To perform the actions described in this article, you must have the Samba software installed. Some actions require that you have a working Server Message Block (SMB)/Common Internet File System (CIFS) network at your disposal.


Understanding TDB files

About the elective LPI-302 exam

Linux Professional Institute Certification (LPIC) is like many other certifications in that different levels are offered, with each level requiring more knowledge and experience than the previous one. The LPI-302 exam is an elective specialty exam in the third level of the LPIC hierarchy and requires an advanced level of Linux system administration knowledge.

To get your LPIC level 3 (LPIC-3) certification, you must pass the two first-level exams (101 and 102), the two second-level exams (201 and 202), and the LPIC-3 core exam (301). After you have achieved this level, you can take the elective specialty exams, such as LPI-302.

Samba stores a lot of information at run time, from local passwords to a list of clients it is expecting to hear from. Some of this data is short lived and can be discarded when Samba is restarted, but some of it is permanent and should not be lost. The data can either be too large or infrequently accessed to keep in memory, or it should persist across restarts. To address these requirements, the Samba team created the Trivial Database. It is effectively a key-value store, meaning that data is stored and retrieved by means of a unique key and there is no joining of tables as in a relational database. Key-value stores—and TDB in particular—are designed to be fast ways to store data to disk and get it back.

Key-value stores

There are many alternatives to TDB, such as the GNU Database Manager (GDBM), but the Samba project had a special need for multiple processes writing to the database at the same time as well as support for locking internal pieces of data. Therefore, the team built their own database manager and called it the Trivial Database Manager. TDB has been extended to support clustered operation through the Clustered TDB (CTDB) project and is available for use by other projects.

Build your own feed

You can build a custom RSS, Atom, or HTML feed so you will be notified as we add new articles or update content. Go to developerWork RSS feeds. Select Linux for the zone and Articles for the type, and type Linux Professional Institute for the keywords. Then, choose your preferred feed type.

If you are familiar with relational databases such as IBM DB2, MySQL, or PostgreSQL, then you will find key-value stores primitive in comparison. Where a relational database may have a table with several columns, a key-value store effectively has two columns: a key column and a value column. Relational databases use Structured Query Language (SQL) to pull information from multiple tables and columns, but key-value stores are limited to operations on the key column.

Because of their simplicity, key-value stores are useful for a smaller set of jobs but can do those jobs very quickly. A key-value store is basically a hash table on disk, so the location of a piece of data can be guessed without having to consult an index.

The TDB in particular was written to handle many concurrent writers and binary data such as internal data structures. Therefore, the fastest way to store and retrieve this data is in binary format. Binary storage means that the database file can be smaller, and there is no requirement to translate data between different formats as data moves in and out of the database.

TDB files used in Samba

Samba stores its TDB files in a few different places. The easiest way to find these files is to look at the output of smbd -b, which is shown in Listing 1.

Listing 1. Showing the smbd build information
# smbd -b
...
Paths:
   SBINDIR: /usr/sbin
   BINDIR: /usr/bin
   SWATDIR: /usr/share/swat
   CONFIGFILE: /etc/samba/smb.conf
   LOGFILEBASE: /var/log/samba
   LMHOSTSFILE: /etc/samba/lmhosts
   LIBDIR: /usr/lib
   MODULESDIR: /usr/lib/samba
   SHLIBE1T: so
   LOCKDIR: /var/lib/samba
   STATEDIR: /var/lib/samba
   CACHEDIR: /var/lib/samba
   PIDDIR: /var/run
   SMB_PASSWD_FILE: /var/lib/samba/private/smbpasswd
   PRIVATE_DIR: /var/lib/samba/private

Several of the paths in Listing 1 point to locations with TDB files. Fortunately, many are the same, which reduces the places you need to look. For example, LOCKDIR, STATEDIR, and CACHEDIR all point to /var/lib/samba, and the PRIVATE_DIR is in a subdirectory called private. You can go into those directories and see the files for yourself.

TDB files can be broadly classified into two groups: those that are persistent and those that are temporary. Table 1 shows the names and functions of the persistent files.

Table 1. Persistent TDB files
File namePurpose
account_policy.tdbStores account policies such as lockout thresholds and password lengths
group_mapping.tdbStores the relationship between Windows NT user identifiers (SID) and UNIX® userids
ntdrivers.tdbStores information about printer drivers
ntforms.tdbStores information about a print queue's forms—for example, paper sizes
ntprinters.tdbStores information about a printer's initialization settings
passdb.tdbUsed only in some security modes to store authentication credentials
registry.tdbA skeleton registry that can be accessed by other Windows systems
secrets.tdbStores local secrets such as the workstation's machine key and credentials like Lightweight Directory Access Protocol (LDAP) passwords
share_info.tdbStores access control list (ACL) information for each local share
winbindd_idmap.tdbStores mapping information between Windows NT SIDs and dynamically created users, if you are using winbind

The TDB files shown in Table 1 should be backed up. The procedure for doing so is provided later in this article.

In addition, there are many temporary TDB files that store state information and cached data. It is not necessary to back them up, because they are rebuilt when Samba starts.


Using TDB files

Samba comes with three tools for manipulating TDB files:

  • tdbdump: Print the contents of a TDB file.
  • tdbbackup: Back up and validate TDB files.
  • tdbtool: Create, view, and modify TDB files.

Peeking inside a TDB file

The fastest way to see what's in a TDB file is to dump it with the tdbdump command. Listing 2 shows a dump of the ntprinters.tdb file.

Listing 2. Using tdbdump on a TDB file
[root@bob ~]# tdbdump /var/lib/samba/ntprinters.tdb 
{
key(21) = "GLOBALS/c_setprinter\00"
data(4) = "\00\00\00\00"
}
{
key(13) = "SECDESC/test\00"
data(140) = "\80\00\00\00\00\00\02\00\80\00\00\00\01\00\04\80\14\00\00\00$\00\00
\00\00\00\00\004\00\00\00\01\02\00\00\00\00\00\05 \00\00\00 \02\00\00\01\02\00\00
...
\00 \02\00\00\00\02\18\00\0C\00\0F\10\01\02\00\00\00\00\00\05 \00\00\00 \02\00\00"
}
{
key(17) = "SECDESC/cups-pdf\00"
data(140) = "\80\00\00\00\00\00\02\00\80\00\00\00\01\00\04\80\14\00\00\00$\00\00
\00\00\00\00\004\00\00\00\01\02\00\00\00\00\00\05 \00\00\00 \02\00\00\01\02\00\00
...
\00 \02\00\00\00\02\18\00\0C\00\0F\10\01\02\00\00\00\00\00\05 \00\00\00 \02\00\00"
}

From the output of Listing 2, you can see that there are three keys in the database. The first key is 21 bytes long (the byte length is shown in parentheses [()]) and is called GLOBALS/c_setprinter followed by a NULL, which is ASCII zero. Non-printable characters are shown in hexadecimal format, which is a backslash followed by two hexadecimal characters. The value for the first key is 4 bytes long and it is entirely NULLs.

The next two keys are called SECDESC/test and SECDESC/cups-pdf and are both NULL terminated. The data is entirely binary and unprintable, so it has been printed out with the non-printable characters in hexadecimal format.

Backing up and restoring TDB files

Table 1 showed several TDB files that persisted across reboots and that should be backed up. Like most databases, you can't just copy the file, because the copy might become corrupted. Backup corruption happens when you copy a file as it is being written so that you have a backup that is in an inconsistent state. One alternative would be to shut down the Samba daemon, and then copy the files.

The easiest way to back up TDB files is to use the tdbbackup utility that comes with Samba. This utility safely copies a TDB file even if it is actively being written to. Another helpful feature of tdbbackup is that it can verify that a TDB file is free of corruption and automatically revert to a backup file if corruption was found. Listing 3 shows a TDB file being backed up.

Listing 3. Backing up a TDB file
[root@bob samba]# ls -l account_policy.*
-rw------- 1 root root 8192 Apr  7  2008 account_policy.tdb
[root@bob samba]# tdbbackup account_policy.tdb 
[root@bob samba]# ls -l account_policy.*
-rw------- 1 root root  8192 Apr  7  2008 account_policy.tdb
-rw------- 1 root root 36864 Dec  8 21:42 account_policy.tdb.bak
[root@bob samba]# tdbdump account_policy.tdb | md5sum
53ea608f0d93061480549c511756b778  -
[root@bob samba]# tdbdump account_policy.tdb.bak | md5sum
53ea608f0d93061480549c511756b778  -

The first command in Listing 3 simply lists all the files starting with account_policy to confirm that there is only one copy. Next, the account policy database is backed up by running tdbbackup account_policy.tdb. The third command looks for any files starting with account_policy to show that a new file with a .bak extension has been created. The size of the file is different from the original, but dumping each file and calculating the MD5 hash shows that their checksums are the same. The larger file size is therefore not a problem, as the contents of each key-value pair are identical.

If for some reason the original account_policy.tdb file were to become corrupted, such as during an unclean shutdown of the system, you can restore the old file from its backup. Listing 4 shows this procedure.

Listing 4. Verifying and restoring a TDB file
[root@bob samba]# ls -l account_policy.tdb
-rw------- 1 root root 1213 Dec  8 21:49 account_policy.tdb
[root@bob samba]# tdbbackup -v account_policy.tdb
tdb_oob len 1256 beyond eof at 1213
restoring account_policy.tdb
[root@bob samba]# ls -l account_policy.tdb*
-rw------- 1 root root 36864 Dec  8 21:49 account_policy.tdb
-rw------- 1 root root 36864 Dec  8 21:42 account_policy.tdb.bak
[root@bob samba]# tdbbackup -v account_policy.tdb
account_policy.tdb : 17 records
[root@bob samba]# tdbdump account_policy.tdb | md5sum
53ea608f0d93061480549c511756b778  -

The first command in Listing 4 shows that the file size has been cut significantly. The tdbbackup command is run again but with the -v flag, which verifies a TDB file. If the file is corrupted, you will see a summary of the errors followed by a note that the file has been restored. Comparing the file sizes with the ls command, you see that the backup was used to replace the current database.

You can safely run the tdbbackup command several times. Running it on a valid database file, like the penultimate command in Listing 4, gives a count of the records in the file. The MD5 sum also matches what was seen before the corruption.

Note: The tdbbackup command accepts wildcards, so you can back up and verify several files at the same time.

Changing TDB files

The tdbtool utility lets you make changes to the data inside a TDB file. This utility also inspects the individual keys and values within the file rather than having to dump the entire file and sort through the output.

Caution

When dealing with Samba, there aren't many cases where you need to change the data within a TDB file. Either the file contains internal data structures or there are better tools to do the job. However, this section is an exam objective, so it's fair game on the test.

tdbtool can accept commands on the command line, or you can open an interactive console. To accomplish your task on the command line, run tdbtool example.tdb command options, where example.tdb is the name of your file, command is the command, and the options to the command go at the end. To use the tdb shell, just run tdbtool by itself or pass the name of the file on the command line.

To create a database, run tdbtool by itself, and then enter create test.tdb. Doing so creates a database on disk called test.tdb and opens it so that any changes you make in this session will be on that file. If you have an existing TDB file, you can specify that name on the command line or use the open command. Curiously, the only way to create a TDB file directly from the command line involves specifying the name twice, such as tdbtool test.tdb create test.tdb, which returns an error but successfully creates a database.

When you have created the database or opened an existing one, the following options are available to you:

  • dump: Show everything in the database, just like tdbdump.
  • keys: Show just the keys in ASCII or, with the hexkeys command, show them in hexadecimal format.
  • erase: Delete everything in the database without prompting.
  • info: Show a summary of the number of keys and bytes used by the database.
  • check: Check to see whether the database is valid.
  • speed: Perform tests to see how fast data can be read and written to the database.
  • show key: Print the value stored with the key.
  • delete key: Delete the key and the value.

To add and manipulate data, you have the insert and store commands. Each accepts parameters for a key and a value, with the value being optional. If you don't specify a value, you will see the key with zero bytes of data are stored in the value.

Inserting a record means to create a new one, while storing a record can overwrite an existing entry. Listing 5 shows the difference between the two commands.

Listing 5. Storing vs. inserting a record
tdb> insert mykey myvalue
tdb> insert mykey newvalue
insert failed
tdb> store mykey newvalue
Storing key:

key 5 bytes
mykey
data 8 bytes
[000] 6E 65 77 76 61 6C 75 65                           newvalue
tdb> store newkey someothervalue
Storing key:

key 6 bytes
newkey
data 14 bytes
[000] 73 6F 6D 65 6F 74 68 65  72 76 61 6C 75 65        someothe rvalue

The sequence of events in Listing 5 is:

  1. Insert a key called mykey with a value of myvalue. This operation succeeds.
  2. Insert a key with the same name as before but with a new value. This operation fails, because the key already exists.
  3. Store the key instead of inserting it. This operation succeeds, and you are given more verbose output.
  4. Store a new key with a new value. This operation succeeds even though the key doesn't exist.

You can use transactions in the tdb shell to allow you to run a series of commands and apply them as a group or cancel them as a group. Listing 6 shows two transactions.

Listing 6. Using transactions
tdb> transaction_start
tdb> insert somekey somevalue
tdb> show somekey

key 7 bytes
somekey
data 9 bytes
[000] 73 6F 6D 65 76 61 6C 75  65                       somevalu e
tdb> transaction_cancel
tdb> show somekey
fetch failed
tdb> transaction_start
tdb> insert somekey somevalue
tdb> transaction_commit
tdb> show somekey

key 7 bytes
somekey
data 9 bytes
[000] 73 6F 6D 65 76 61 6C 75  65                       somevalu e

The transaction is first started with the transaction_start command. Next, a key is inserted. If another process were to be reading the database, it would not see this key, because it is not committed. The process with the transaction open does see the key. The transaction is then canceled with transaction_cancel. The key no longer appears.

The process is then re-run, but the transaction is committed with transaction_commit. The key then exists to all readers.

Other readers may be blocked if you are in a transaction, which means that they will hang until the transaction is finished. So be careful when using transactions on production databases! They are a great safety feature but can potentially hurt performance if overused.


Beyond TDB

User accounts can be stored in a variety of places depending on your configuration, and two tools give you a command-line interface while taking care of the back-end communication. For example, you can use the same commands even if your user data is stored in LDAP instead of TDB files.

Using smbpasswd

The smbpasswd utility can add and delete user or machine accounts and change passwords. It is most commonly used to perform the latter task, either to change the current user's password or by root to change another user's password.

In later articles, you will learn about the different password back ends in more depth, but at a high level, the Samba passwords can be stored in several different ways depending on the vintage of Samba and which systems you're integrating with. Both smbpasswd and pdbedit (described next) can perform their actions against any back end, even if it isn't TDB.

Microsoft clients pass passwords around the network as a Microsoft-specific hash rather than plain text or a UNIX password hash. This means that it is not possible to take a Microsoft password hash and determine whether the password is the same as the password stored in the UNIX password database. Therefore, Samba must keep a separate password database for the Microsoft hashes; this is referred to as the password back end.

Using pdbedit

pdbedit manages the Samba user database and the account policies. It can do anything that smbpasswd can plus manage policies and migrate accounts between different back ends.

To show all the users in the database, run pdbedit -L. You can get even more details about your users by passing the -v flag, which is shown in Listing 7.

Listing 7. A verbose listing of users
[root@bob tmp]# pdbedit -L -v
---------------
Unix username:        sean
NT username:
Account Flags:        [U          ]
User SID:             S-1-5-21-2287037134-1443008385-640796334-1001
Primary Group SID:    S-1-5-21-2287037134-1443008385-640796334-513
Full Name:            Sean
Home Directory:       \\bob\sean
HomeDir Drive:
Logon Script:
Profile Path:         \\bob\sean\profile
Domain:               BOB
Account desc:
Workstations:
Munged dial:
Logon time:           0
Logoff time:          never
Kickoff time:         never
Password last set:    Mon, 24 May 2010 21:28:49 CDT
Password can change:  Mon, 24 May 2010 21:28:49 CDT
Password must change: never
Last bad password   : 0
Bad password count  : 0
Logon hours         : FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

You must be logged in as root for the pdbedit command to work. You can edit various user settings, such as last login time, password changes, and home directories, as if you were using the Microsoft utilities.

Each user parameter has its own command-line option, so consult the pdbedit(8) manpage for the specifics.

It is also important to note that Samba treats machine accounts and user accounts in a similar manner. Listing 8 shows the password database from a Samba server configured as a domain controller.

Listing 8. The password database, including machine accounts
[root@sergeant ~]# pdbedit -L
root:0:root
mythupstairs$:4294967295:MYTHUPSTAIRS
BOB$:1043:Machine
sean:1002:Sean,,,
sergeant$:4294967295:Machine

The names ending with a dollar sign ($) are machine accounts and are used to authenticate a machine to the domain. The corresponding secret would be stored in secrets.tdb on the remote server and passdb.tdb on the domain controller.


Moving forward

This is the end of the Concepts, Architecture, and Design topic. In the next article, you will begin Topic 311 by learning how to download the Samba source code and compile it.

Resources

Learn

Get products and technologies

  • Download Samba, and keep on top of the latest developments to the software.
  • Download the TDB source code if you're interested in using TDB in one of your projects.
  • Look at account-management tools for Samba if you need a graphical interface or better integration with different environments.
  • Evaluate IBM products in the way that suits you best: Download a product trial, try a product online, use a product in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement Service Oriented Architecture efficiently.

Discuss

  • Get involved in the My developerWorks community. Connect with other developerWorks users while exploring the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Linux on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux
ArticleID=631383
ArticleTitle=Learn Linux, 302 (Mixed environments): Trivial Database files
publish-date=03082011