Tealeaf Support Troubleshooting FAQ

This section contains frequently asked questions of Customer Support on how to administer the Tealeaf® CX system. Use the links below to jump to a question and the related answer.

If you have a question that is not answered in this section, you may use the Send Feedback link at the bottom of the page to submit your question to Tealeaf.

Problem - I changed my privacy rules to scrape or block data, but they do not seem to be working. What should I check?

What to look for first

Did you restart the Tealeaf Transport service? This must be completed on each Tealeaf processing server where privacy changes occurred. If you modified privacy using our TMS functionality, it should offer to restart this service while applying the configuration file change. This action can be validated by checking the Windows Event Application Log on the server(s) in question.
If the service restarted successfully but the rule is not working, more than likely you find an error log. Each time the Tealeaf Transport Service restarts the privacy rules are loaded in an Windows Event Application log entry. Details of any problem should be available.

If you file a case with Customer Support

Note and report any visible error messages in relation to the privacy rules as they were loaded by the Tealeaf Transport Service restart.
Attach the following:
- Privacy.cfg
- Example session data (can be in .txt document format) along with an explanation of what you are trying to do

Problem - The Tealeaf Transport Service will not start or will not stay running

What to look for first

Have any changes been made immediately prior? Especially in the case of privacy.cfg changes, try disabling new rules and try again.
Review the Windows Event Application Log on the server(s) in question.

If you file a case with Customer Support

Note and report any visible error messages in relation to the privacy rules as they were loaded by the Tealeaf Transport Service restart.
Attach the Windows Application Event Log (exported as an EVT file, zipped)
If there is a TealeafCaptureSocket.exe-(vW.X.YY.ZZZZ)-YYYYMMDD-HHMMSS*.dmp file in the Tealeaf/logs folder, include it.

Problem - I am receiving a portal status error: "Sessions waiting to be indexed broken its threshold"

What to look for first

If you did not immediately respond to this error message, log in to the Tealeaf portal and browse to the Tealeaf - System Status - Canister report and select the canister reporting indexing issues. Check for the "Sessions waiting for archive to disk". If this number is very high and not consistently decreasing, it indicates a problem.
If your canister/indexes are stored on a SAN as opposed to local storage, have your SAN team confirm disk throughput is as expected. In many cases, the root cause of this issue is inadequate disk performance.

If you file a case with Customer Support

Attach the following logs: CSS_1966_<CANISTERNAME>CS<DAYSDATE>.txt, CSS_1966_<CANISTERNAME>DL<DAYSDATE>.txt, CSS_1966_<CANISTERNAME>CS<DAYSDATE>.txt where <CANISTERNAME> is the name of your canister server and <DAYSDATE> is the date (yyyymmdd) on which you are having the issue.

Problem - I have errors indicating indexing failed, crashed, stopped or deleted while cleaning up session data.

What to look for first

Fortunately, indexes are one of the recoverable file types in the Tealeaf system and they can be re-created. To do this:
- Log in to the Tealeaf canister in question, open a browser and navigate to http://localhost:19000/CIC (if prompted for authentication, the user and password are ssadmin/ssadmin).
- If you are uncertain whether sessions are missing from the index, you can compare the two columns to ensure the session count and indexed document count match.
- If the columns match but you appear to be having issues or suspect index corruption, you can perform a "check indexes" from the top menu.
- Finally, if you'd like the system to do a full audit of indexes and repair any problems that it finds, select "check and fix". Depending on the number of sessions and the number of days' data you collect, this could take up to a full day to complete.

If you file a case with Customer Support

If you have performed the steps above and the indexing service does not continue running, gather any relevant errors from the Windows Event Application log and attach them to the support case.

Problem - Searching for active sessions returns no results

What to look for first

If your search returns no results, first change Limit Hits To: <No Limit>. If results return, it means your method of Tealeaf sessionization was broken. Confirm that the expected cookie is not being found by checking that the TLMERGEID is blank in one of the request buffers. The cookie being referenced should be in parentheses. At this point, you must identify another unique cookie that can stitch together pages or determine where the existing cookie has gone.
If there are still no results, you should open the Tealeaf Pipeline Status utility either locally on the server or in TMS (available in later versions).
If you confirm that traffic is continuing to flow through the pipeline (usually the canister session agent is last), then sessions should still be making it into the active canister. At this point, there is a possible permission issue and depending on your form of authentication - portal or NT, you should contact Tealeaf Support

If you file a case with Customer Support

Describe the steps that are previously attempted.
Attach a TLS file that contains a representative session with the problem session IDs.

Problem - Searching completed sessions returns no results

What to look for first

Check the completed session template for "available dates". The dates available should equal the total number of days you expect to retain production data.
If you have access to the Tealeaf Canister, log in, browse to http://localhost/CIC (user and password are ssadmin/ssadmin) and compare the two columns to ensure the session count and indexed document count match. If the indexes are unavailable the sessions are not searchable. If the indexes are gone, it's possible to re-create them with the check and fix command. See: Problem: I have errors indicating indexing failed or crashed.
If indexes are available and days appear to be present, confirm that there are no data segmentation filters in place that would be excluding content. This can be done by opening "searchconfig" (without the quotes) from the Windows - Start - Run menu, and selecting modify under Domain Local Groups. In the resulting window, there is data segmentation filters at the bottom. If someone assigned a filter to a group and the account searching is a member of this group, it will append this to any search and may result in no results found.

If you file a case with Customer Support

Note and report any visible error messages from the failed search and the results of initial troubleshooting above.
Examine the Windows Event Application Log for errors.
Collect and attach the TLSrchSrv.<DaysDate>.log

Problem - I am trying to search using "and same page" and am having troubles

What to look for first

"And same page" does not work with "does not include" logic. If any of your search criteria include this, it returns no results.
If searching for more than two things on the same page, try reducing the search terms one at a time to see if you get results.
If you notice that the search results reflect x number of y sessions where y is greater than x, this indicates Tealeaf found y number of sessions with your search terms in the same session, and then processed those sessions to determine x, where x is the number of sessions reflecting "and same page" logic. This is expected functionality, not a reflection of missing sessions.

If you file a case with Customer Support

Note and report any visible error messages from the failed search and the results of initial troubleshooting above.
Collect and attach the TLSrchSrv.<DaysDate>.log.

Problem - It says I have only three days' data, but I should have 14; what do I check?

What to look for first

Within the portal, navigate to the Tealeaf - System Status - Storage report, change to each individual Storage Server, and determine the number of days' worth of data that appears.
Locally on the canister/processing server, navigate to the Canister.dbs directory, sort by type, and check the .dat files for: LSSN_<daysdate>_<canistername>.dat. There is one .dat file for each canister day. If you confirm the expected dates are available, then they are not being recognized. A canrebuild should address this:
- Windows Start - Run - canrebuild
- Important: Ensure that Preserve session data is CHECKED, and complete the canrebuild.
- Return to the portal and confirm that the dates are restored.

If you file a case with Customer Support

Report steps that are taken in troubleshooting.
Check the Windows Event Application Log for errors that are related to Tealeaf.
Attach the TLTMaint.log file.

Problem - Events not appearing in the portal

What to look for first

If you created an event and it is not displaying anywhere in the portal make sure to check both the search templates and event charts.
In the event itself, double-check that "Display event in portal" is checked and that Building blockis not checked. Both intentionally hide the event in the portal.
Confirm that the event is not a session-level event and that you are not looking for it in an active search. Session-level events are evaluated when an active session is closed and written out as a completed session. As a result, session-level events cannot be found in active sessions.

If you file a case with Customer Support

Attach a screen capture of the configured event

Problem - Event data is not current, or is missing entirely

What to look for first

Event data is collected by the Tealeaf Data Collector service on the portal server. At 5-minute intervals, it communicates with each of the canisters to become aware of new events and to gather statistics about these events for import into SQL. If data is missing for hours, it likely means the data collector service has been unable to communicate with the canister.
Check the services on the Tealeaf Portal server and ensure that the Tealeaf Data Collector service is running.
Examine the Windows Event Application Log for errors. If you find an error mentioning a timeout regarding either the SQL server or the canister itself, you can alter either or both timeout settings in the portal to allow more time for this process to complete:
- Under the Tealeaf - Portal Management - CX Settings - Data Collector menu you find two values: Canister connection timeout (seconds), and Database Connection timeout(seconds).
- The former can be increased in an attempt to address timeouts to the canister, while the latter can be raised to accommodate performance issues in relation to the import of statistics into SQL.

If you file a case with Customer Support

Attach the following:
- TLDataCollector.log (if data collection stopped in the current day); or
- TLDataCollector_<daysdate>.log (where <daysdate> reflects the date of the last hour of event reporting data.

Problem - Alerts not firing and/or emails not arriving

What to look for first

If you feel certain that alerts should be firing and sending email, there is an alert report in the portal under Active - Alert Monitor. If you find alerts in an alert state, the specified action in the alert should be occurring.
Confirm that the alert is configured to send to valid email addresses and that Email is checked.
In the Tealeaf Portal, under Tealeaf - TMS, navigate to the reporting server, expand the Alert Service, select Alert Service Configuration, click View/Edit, and ensure an Email From Addressis configured. Most SMTP servers do not require that the sender is a real email account, only that it have a correct domain prefix, for example @yourcompany.com.

If you file a case with Customer Support

Note any troubleshooting steps taken.
Attach the following:
- TLAlertSrv_<daysdate>.log (where <daysdate> reflects the date you expect to see the alert email).

Problem - Cannot log in to the Portal

What to look for first

Are you using NT or portal authentication? If you normally type a password, it's Portal authentication. If you normally log in without a password, it is NT authentication.
Confirm that the Tealeaf SQL Server database is operational and accessible over the network to the portal server.
Try logging in with the master "admin" account.
Confirm that the Tealeaf Search Service is running on all Tealeaf Window Servers.

If you file a case with Customer Support

Report any errors displaying in the portal.
Examine the Windows Event Application log for error messages and attach to case.

Problem - No hits in the canister

What to look for first

Portal Status Report: Are all servers up? Is there a working connection to the Passive Capture Application server?
Portal Status Report: Has DecoupleEx commenced queuing? If yes...
- Is the canister using too much memory?
- Is the canister running out of disk space?
Capture Server Web Console: Any warning indicators?
Is the span port or load balancer oversubscribed?

If you file a case with Customer Support

Note and report any visible error messages.
Determine as precisely as possible the date and time when the problem first appeared.
Attach the following:
- Portal Status Report
- Capture server maintenance log
- Capture server capture log
- Capture server error log
- Capture server full-day statistics file

Problem - Many one-hit sessions

What to look for first

Are there IP addresses that the capture server should filter out?
Is the canister session timeout too short?

If you file a case with Customer Support

Note as precisely as possible the date and time when the problem first appeared.
Attach the following:
- Portal Status Report
- Capture server capture log