Troubleshooting package dependency problems with Resilient Circuits for IBM QRadar SOAR

Troubleshooting

Problem

When updating pip installed Python packages, you might come across dependency problems. While the problem is with pip, which is not an IBM product, this document tries to provide guidance for clients who come across such situations.

Symptom

You might see an error returned by using pip to install a package or you might find that the upgrade processed but Resilient Circuits does not start or starts in a loop.

Cause

Packages such as resilient, resilient-circuits, resilient-lib or resilient-sdk have dependencies on other packages. Upgrading versions of these IBM packages, the dependencies of the new version might be different to the previous version installed. In this case, pip tries to install all the dependant packages to fulfill the new package's requirements.

Occasionally, pip tries to upgrade a version of a package but the package is required by another package. Pip does not upgrade the IBM package, or it continues with the upgrade breaking another package.

Diagnosing The Problem

If you find that Resilient Circuits does not start after an upgrade of packages by using pip, follow MustGather: Collecting logs for IBM Resilient Circuits to gather logs.

From the logs, look at /var/log/messages, which might describe why Resilient Circuits does not start. You might find information in /var/log/messages that is not contained in the app.log when there are dependency problems.

May 29 12:11:34  circuits-server resilient-circuits: 2022-05-29 12:11:34,955 DEBUG [_api] Lock 140025834584888 released on /home/integration/.resilient/resilient_circuits.lock
May 29 12:11:34  circuits-server resilient-circuits: Traceback (most recent call last):
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/bin/resilient-circuits", line 33, in <module>
May 29 12:11:34  circuits-server resilient-circuits: sys.exit(load_entry_point('resilient-circuits==45.0.3150', 'console_scripts', 'resilient-circuits')())
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/bin/resilient_circuits_cmd.py", line 404, in main
May 29 12:11:34  circuits-server resilient-circuits: config_file=args.config_file)
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/bin/resilient_circuits_cmd.py", line 85, in run
May 29 12:11:34  circuits-server resilient-circuits: app.run(**kwargs)
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/app.py", line 231, in run
May 29 12:11:34  circuits-server resilient-circuits: application = App(*args, **kwargs)
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/app.py", line 70, in __init__
May 29 12:11:34  circuits-server resilient-circuits: self.do_initialization()
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/app.py", line 111, in do_initialization
May 29 12:11:34  circuits-server resilient-circuits: self.component_loader = ComponentLoader(self.opts)
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/component_loader.py", line 65, in __init__
May 29 12:11:34  circuits-server resilient-circuits: installed_components = self.discover_installed_components()
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/resilient_circuits/component_loader.py", line 97, in discover_installed_components
May 29 12:11:34  circuits-server resilient-circuits: cmp_class = ep.load()
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2464, in load
May 29 12:11:34  circuits-server resilient-circuits: self.require(*args, **kwargs)
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2487, in require
May 29 12:11:34  circuits-server resilient-circuits: items = working_set.resolve(reqs, env, installer, extras=self.extras)
May 29 12:11:34  circuits-server resilient-circuits: File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 782, in resolve
May 29 12:11:34  circuits-server resilient-circuits: raise VersionConflict(dist, req).with_context(dependent_req)
May 29 12:11:34  circuits-server resilient-circuits: pkg_resources.ContextualVersionConflict: (six 1.16.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('six==1.14.0'), {'mail-parser'})
May 29 12:11:35  circuits-server systemd: resilient_circuits.service: main process exited, code=exited, status=1/FAILURE
May 29 12:11:35  circuits-server systemd: Unit resilient_circuits.service entered failed state.
May 29 12:11:35  circuits-server systemd: resilient_circuits.service failed.
May 29 12:11:35  circuits-server systemd: resilient_circuits.service holdoff time over, scheduling restart.
May 29 12:11:35  circuits-server systemd: Stopped Resilient-Circuits Service.
May 29 12:11:35  circuits-server systemd: Started Resilient-Circuits Service.

This content from /var/log/messages indicates that six 1.16.0 is installed, but the package mail-parser has a requirement for six==1.14.0. This dependency problem stops Resilient Circuits from starting.

$ resilient-circuits config -u -l fn-mitre-integration

Error:
UPDATING config file /home/integration/.resilient/app.config
Traceback (most recent call last):
File "/usr/local/bin/resilient-circuits", line 10, in sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/resilient_circuits/bin/resilient_circuits_cmd.py", line 679, in main
generate_or_update_config(args)
File "/usr/local/lib/python2.7/site-packages/resilient_circuits/bin/resilient_circuits_cmd.py", line 241, in generate_or_update_config
func = entry.load()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2442, in load
self.require(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2465, in require
items = working_set.resolve(reqs, env, installer, extras=self.extras)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (six 1.12.0 (/usr/local/lib/python2.7/site-packages), Requirement.parse('six>=1.13.0'), set(['stix2']))

This output describes that six 1.12.0 is installed but stix2 requires six to be 1.13.0 or greater.

Resolving The Problem

Identifying the problem can be time consuming depending on the number of packages installed. IBM does not restrict what packages a client can install on their Integration Server. The Integration server operating system, pip, and Python are the responsibility of the client as they are not IBM applications.

While this document provides guidance on where to look it does not provide a total solution. If you cannot resolve these dependencies, you might need to engage your internal support teams or use other resources such as pip forums. You can also request assistance from the IBM QRadar SOAR community.

Investigation and resolution

May 29 12:11:34  circuits-server resilient-circuits: pkg_resources.ContextualVersionConflict: (six 1.16.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('six==1.14.0'), {'mail-parser'})

Running pip freeze reveals that mail-parser 3.12.0 is installed (https://pypi.org/project/mail-parser/3.12.0/#files)

The contents of requirement.txt for that version of the package is

ipaddress==1.0.23
simplejson==3.17.0
six==1.14.0

The installed version of mail-parser defines that six must be version 1.14.0

The output of pip freeze showed that an old version of an IBM function is installed. The older function specifies that mail-parser must be 3.12.0. Later versions of the same function changed the requirements so that it selects the latest version of the package, greater than or equal to 3.15.0, but still in the 3.* code stream.

mail-parser~=3.15

Installing the latest version of the function updated mail-parser to version 3.15.0 or greater.

Mail-parser 3.15.0 - https://pypi.org/project/mail-parser/3.15.0/#files

The contents of requirement.txt for that version of the package is

ipaddress>=1.0.23; python_version < '3.3'
simplejson>=3.17.0
six>=1.14.0

The package six does not need to be 1.14.0, unlike with mail-parser 3.12.0. With mail-parser 3.15.0, six must be greater or equal to 1.14.0. This means when mail-parser is updated, six 1.16.0, which is already installed, works with mail-parser. The dependency is fulfilled and Resilient Circuits starts.

If you want to identify the requirements for particular functions installed on the Integration Server, you can check the requires.txt. Taking fn_outbound_email as an example.

$  pip show fn-outbound-email
Name: fn-outbound-email
Version: 1.3.0
Summary: Resilient Circuits Components for 'fn_outbound_email'
Home-page: https://github.com/ibmresilient/resilient-community-apps/tree/master/fn_outbound_email
Author: IBM Resilient
Author-email:
License: MIT
Location: /home/integration/.pyenv/versions/3.6.9/envs/python369/lib/python3.6/site-packages
Requires: Jinja2, resilient_circuits, resilient_lib, six
Required-by:

$ ls -l /home/integration/.pyenv/versions/3.6.9/envs/python369/lib/python3.6/site-packages/fn_outbound_email-1.3.0-py3.6.egg-info
total 28
-rw-rw-r--. 1 integration integration 1 Feb 11 14:33 dependency_links.txt
-rw-rw-r--. 1 integration integration 407 Feb 11 14:33 entry_points.txt
-rw-rw-r--. 1 integration integration 1187 Feb 11 14:33 installed-files.txt
-rw-rw-r--. 1 integration integration 370 Feb 11 14:33 PKG-INFO
-rw-rw-r--. 1 integration integration 67 Feb 11 14:33 requires.txt
-rw-rw-r--. 1 integration integration 718 Feb 11 14:33 SOURCES.txt
-rw-rw-r--. 1 integration integration 18 Feb 11 14:33 top_level.txt

$ cat /home/integration/.pyenv/versions/3.6.9/envs/python369/lib/python3.6/site-packages/fn_outbound_email-1.3.0-py3.6.egg-info/requires.txt
resilient_circuits>=39.0.0
resilient_lib>=32.0.0
Jinja2>=2.9.6
six

The same information is available in the function's setup.py, which you can view in IBM's GitHub page -> https://github.com/ibmresilient/resilient-community-apps/blob/master/fn_outbound_email/setup.py

    install_requires=[
        'resilient_circuits>=39.0.0',
        'resilient_lib>=32.0.0',
        'Jinja2>=2.9.6',
        'six'

App Host

IBM QRadar SOAR App Host avoids dependency problems by running each function inside its own container. For complex deployments of packages, App Host might be a consideration.

Resilient Circuits on App Host

It is not recommended that you install Resilient Circuits on an App Host. In later version of App Host, from 1.9, some packages that were traditionally managed by pip are now managed by yum. This causes problems when pip attempts to uninstall and reinstall these packages in line with the requirements of other packages.

If you wish to run Resilient Circuits, then provision an integration server and use the documented approaches to installing.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSA230","label":"IBM Security QRadar SOAR"},"ARM Category":[{"code":"a8m0z0000008ZtGAAU","label":"Integrations-\u003EResilient Circuits"}],"ARM Case Number":"TS009507828","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"},{"Type":"MASTER","Line of Business":{"code":"LOB24","label":"Security Software"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEGM63","label":"IBM Security QRadar SOAR on Cloud"},"ARM Category":[{"code":"a8m0z0000008ZtGAAU","label":"Integrations-\u003EResilient Circuits"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Tips

Troubleshooting package dependency problems with Resilient Circuits for IBM QRadar SOAR