IBM Security Information Queue FAQ

Answer

A1. The latest version of ISIQ can always be found on its starter kit web page: https://www.ibm.com/support/pages/ibm-security-information-queue-starter-kit

A2. Not in the sense of being a specialized point-to-point solution like ISIGADI. ISIQ is a general-purpose data exchange broker. It consists of several services running in separate containers on a single docker node or on a cluster of nodes. One of ISIQ's first use cases is to exchange data between ISIM and IGI, but that's just one use case.

A3. Yes, you can execute all ISIQ containers on a swarm with one docker node. However, you risk loss of availability if there's a failure on the host system of that node.

A4. Docker Community Edition (CE) 19.03 running on CentOS, Debian, Fedora, or Ubuntu with x86_64/amd64, 32 GB RAM, 4 vCPU, 100 GB free disk space.

A5. There are challenges to this approach. For example, if you use the optional ISIQ logs stack, it includes logspout, which reads in the docker logs of all containers on the host and sends them to logstash. If you're running many other containers, it could create a logging bottleneck. There are configuration options that might help, but for now we suggest a dedicated docker server instance for ISIQ.

A6. We recommend you specify an odd number of hosts (which is why the ISIQ-supplied YAML files assume a three-node cluster) since distributed algorithms use majority elections.

A7. Run the command `docker swarm init` to initialize the swarm, and then run `docker swarm join` to add extra hosts. Some ISIQ services need to maintain state via docker volumes. In a cluster, you must either pin those services to a particular host, or else set up a shared file system so that state data can be read from whichever node the service starts on. For more information about this subject, see the "Stateful Services" section of the ISIQ Deployment Guide.

A8. Yes, customers have successfully deployed ISIQ docker swarms on AWS. If you're implementing a cluster configuration, be sure to set your AWS firewall rules to satisfy ISIQ's inter-container networking requirements. See the "Port Usage" and "Docker Swarm Management" sections of the ISIQ Deployment Guide.

A9. There are configuration assumptions in ISIQ's use of Kafka that might not match your existing environment.

Examples:

a) ISIQ uses a naming convention for isolating topics by product. ISIQ topic names could end up clashing with your other topic names.

b) Although ISIQ relies on TLS connections behind docker's network fabric, ISIQ does not use TLS between stack layers, which might be inconsistent with your security configuration.

The upshot: This capability is probably out of scope. If you want to exchange data with existing Kafka deployments, we suggest you wait until ISIQ delivers replication technology.

A10. In principle, ISIQ can use any provider that supports OIDC. We tested quite a few in-house, but not all. Since every provider we tried (IGI, Google, ISAM, Azure AD, IBM Security Verify) worked, we expect others should work too.

A11. Yes, you can use the IGI admin provider to authenticate IGI administrators to ISIQ. If feasible, we recommend ISAM or IBM Security Verify for enterprise-level authentication. For more information about this subject, see the "OpenID" section of the ISIQ Deployment Guide.

A12. In ISIQ, when you delete a configured product such as IGI, any topic data associated with that product also gets deleted.

A13. Each subscription definition on the ISIQ product dashboard page offers a "Reprocess" button that lets you replay one or more source topics from the beginning. This option can be used to ensure your producer and consumer are in sync. For more information, see the "Reprocessing Topics" section of the ISIQ User's Guide.

A14. ISIQ offers a validation tool to automate synchronization checking. For more information, see "Appendix H: Optional ISIQ Tools" in the ISIQ User's Guide.

A15. Yes, this elastic computing style should work, given due caution against losing state for stateful services (Zookeeper, Kafka broker, time-series database) when you do the removal. In other words, make sure that replication has time to create data replicas before you bring down all copies. Also, don't remove nodes that hold the only copy of data. Stateless service replicas will easily be moved among remaining nodes. Again, refer to the "Stateful Services" section of the ISIQ Deployment Guide.

A16. Default retention is set according to "log compaction" (see http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html for an explanation). Simply put, it means that after a sufficient time period, only the last state of each object remains in the queue. For state data, this behavior makes sense because at some future time you might want to replay the data or introduce a new consumer/subscriber. For event-oriented data such as changelog data, you can use command-line tools to configure a time-based retention policy. We are considering adding APIs and console features in ISIQ to enact replays, set retention policies, etc. It's possible to set a minimum compaction lag if needed to give consumers a chance to see all states of an object (note: this isn't necessary for the IGI connector). Otherwise, they will see at least the last state. After a configurable retention period (24 hours by default), Kafka "tombstones" are removed and no object reference remains.

A17. As previously mentioned, under the default log compaction policy, only the last reference to each object is retained. In the example cited, after sufficient time, nothing would be replayed once the object gets compacted. Even tombstones are cleared after all other references are compacted and the retention period (default 24 hours) elapses. Note: depending on volume of activity, other references might get compacted somewhat unpredictably up to a time limit set by Kafka's log.roll configuration (7 days by default). The process happens whenever Kafka goes through partition segment rotation, which occurs when a time or segment size limit is reached.

A18. The ISIQ connector waits before it delivers to IGI an "incomplete" object, specifically, one that's missing data the object depends on. Eventually (as determined by the CONNECT_DEPENDENCY_WAIT_TIMEOUT environment variable value in ISIQ's connect-stack.yml file), if the dependencies still aren't resolved, ISIQ delivers the object to IGI with a stub for the missing data. In that situation, you can implement rules in IGI to attempt a graceful handling of the missing data.

A19. The ISIQ-supplied YAML files contain "logging:" definitions to control the number and size of docker log files per ISIQ stack. These definitions take precedence over any system-wide docker log rotation settings that you configured. For more information, see the "Docker Log Rotation" section of the ISIQ Deployment Guide.