Install the ISAM Agent

Normally you would download the ISAM Agent installer from the Synergex Resource Center but to simplify the workshop a copy of the installer is already present on your Linux server:

~/installers/sip-isam-agent-1.0.0-1.el9.x86_64.rpm

Now it is time to install the Streaming Integration Platform ISAM Agent:

sudo dnf install -y ~/installers/sip-isam-agent-1.0.0-1.el9.x86_64.rpm

On the workshop system you can use the shortcut:

sip install isam

As the installation proceeds you will see messages like this:

===============================================================================
 Streaming Integration Platform: ISAM AGENT
===============================================================================

By installing this product you accept the terms of the Synergex Product License
agreement (PLA) which can be viewed at:

https://www.synergex.com/wp-content/uploads/2022/08/SDE_PLA_CURRENT-1.pdf

  Using configuration data from /etc/sysconfig/sip.conf
   - Service name  : sip-isam-agent
   - User account  : sip-service
   - User group    : sip-service
   - Dflt profile  : default
   - HTTPS port    : 5126

  Installing       : sip-isam-agent-1.0.0-1.el9.x86_64                                                                                                                                                                                                   1/1
  Running scriptlet: sip-isam-agent-1.0.0-1.el9.x86_64                                                                                                                                                                                                   1/1

  Starting service

 The Streaming Integration Platform ISAM AGENT has been installed.

 If you are using Configuration Server and configured the environment before
 starting this installation, and provided information about the environment
 by using /etc/sysconfig/sip.conf then the ISAM agent should already be up
 and running.

 If you are not using Configuration Server you will need to manually provide
 configuration files by creating a directory with the lowercase name of the
 profile that you wish to use in /etc/synergex/sip and also copy the necessary
 configuration files to that directory. The necessary configuration files are:

  - MasterConfig.json
  - ServerConfig-ISAMAgent.json

 If you change anything in the configuration you must restart the service:

    sudo systemctl restart sip-isam-agent

 To check the status of the service:

      sudo systemctl status sip-isam-agent

 To view the service log:

      sudo journalctl -u sip-isam-agent

===============================================================================


  Verifying        : sip-isam-agent-1.0.0-1.el9.x86_64                                                                                                                                                                                                   1/1

Installed:
  sip-isam-agent-1.0.0-1.el9.x86_64

Complete!

Because you are using Configuration Server and pre-configured the environment, the agent should have successfully started.

Confirm the agent is running by viewing its log

sudo journalctl -u sip-isam-agent

On the workshop system you can also use the shortcut:

sip log isam

Scroll back a little and you should see the information that is logged when the agent starts:

Oct 12 13:55:42 sipws systemd[1]: Started Streaming Integration Platform ISAM Agent.
Oct 12 13:55:42 sipws systemd[1]: Stopping Streaming Integration Platform ISAM Agent...
Oct 12 13:55:42 sipws systemd[1]: sip-isam-agent.service: Deactivated successfully.
Oct 12 13:55:42 sipws systemd[1]: Stopped Streaming Integration Platform ISAM Agent.
Oct 12 13:55:42 sipws systemd[1]: Started Streaming Integration Platform ISAM Agent.
Oct 12 13:55:42 sipws bash[5891]: ===============================================================================
Oct 12 13:55:42 sipws bash[5891]: Streaming Integration Platform: ISAM Agent
Oct 12 13:55:42 sipws bash[5891]: ===============================================================================
Oct 12 13:55:42 sipws bash[5891]: Bootstrap mode:         Configuration Server
Oct 12 13:55:42 sipws bash[5891]: Configuration server:   https://sipws:5121
Oct 12 13:55:42 sipws bash[5891]: Configuration cache:    /etc/synergex/sip/default
Oct 12 13:55:42 sipws bash[5891]: Profile name:           default
Oct 12 13:55:42 sipws bash[5891]: --------------------------------------------------------------------------------
Oct 12 13:55:42 sipws bash[5891]: Registering with Configuration Server
Oct 12 13:55:43 sipws bash[5891]: Registered OK
Oct 12 13:55:43 sipws bash[5891]: Requesting authentication token from Configuration Server
Oct 12 13:55:43 sipws bash[5891]: Downloading master configuration from Configuration Server
Oct 12 13:55:43 sipws bash[5891]: Downloading server configuration from Configuration Server
Oct 12 13:55:43 sipws bash[5891]: Downloading TLS certificate from Configuration Server
Oct 12 13:55:43 sipws bash[5891]: Saving CA certificate to trust to local file /etc/synergex/sip/config-server-ca.crt
Oct 12 13:55:43 sipws bash[5891]: --------------------------------------------------------------------------------
Oct 12 13:55:43 sipws bash[5891]: Admin UI port:      5126
Oct 12 13:55:43 sipws bash[5891]: Local database:     /etc/synergex/sip/isam-agent/isam-agent.db
Oct 12 13:55:43 sipws bash[5891]: TLS cert:           /etc/synergex/sip/isam-agent/isam-agent-sipws.pfx
Oct 12 13:55:43 sipws bash[5891]: Kafka brokers:      sipws:9092
Oct 12 13:55:43 sipws bash[5891]: CDC topic:          DEFAULT_CDC/0
Oct 12 13:55:43 sipws bash[5891]: Snapshot requests:  DEFAULT_SNAPSHOT_REQUEST/0
Oct 12 13:55:43 sipws bash[5891]: Snapshot responses: DEFAULT_SNAPSHOT_RESPONSE/0
Oct 12 13:55:43 sipws bash[5891]: S3 provider:        RustFS
Oct 12 13:55:43 sipws bash[5891]: S3 endpoint:        https://sipws:9000
Oct 12 13:55:43 sipws bash[5891]: S3 region:          us-west-2
Oct 12 13:55:43 sipws bash[5891]: S3 bucket:          default
Oct 12 13:55:43 sipws bash[5891]: S3 path:
Oct 12 13:55:43 sipws bash[5891]: S3 access key:      sipagent
Oct 12 13:55:43 sipws bash[5891]: Log file:           /sip/log/isamagent-default-20261012-135543.log

Each time the ISAM Agent starts it checks to make sure that all of the data files that are defined in the configuration metadata are present, and if any are found to be missing, a rehydration operation is triggered to ensure that all of the data is present.

Oct 12 13:55:43 sipws bash[5891]: 20261012-135543 WARNING: The local replica has not been seeded

If the agent determines that the data is not completely present it will bring it up to date by initiating a snapshot restore:

Search the snapshot response topic for the most recent ISAM (counted file) snapshot.
If a snapshot exists, check if the corresponding TAR file is present in S3 storage.
If a TAR file was located, download it, extract it, and load the data files.
If a TAR file was NOT located, send a snapshot request message, wait for a snapshot response message, and restore the new snapshot.

Here you can see that the existing snapshot that you made earlier was located and downloaded:

Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Restoring snapshot 2d35e4a2-5e4b-4e9b-aef0-8cb49bf16813 from SNAPSHOT_0.TAR (snapshot CDC offset 0)
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Expected snapshot didnt exist locally at SNAPSHOT_0.TAR attempting to use s3 client to retrieve it
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Downloaded snapshot from S3 to temporary file /tmp/nllm0wvo.1xy (44814336 bytes)

And then the snapshot was restored, creating new data files:

Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 ------------------------------------------------------------
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Snapshot restore starting. RootFolder=/sip/data, ReplaceFolder=, configured files=9, metadata entries=9, snapshot version=1, snapshot blob=SNAPSHOT_0.TAR
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 ----------------------------------------
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Creating customer.ism from CUSTOMER.CNT.LZ4
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Extract starting to /sip/data/customer.cnt
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Record length: 287 Keys 8 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Load starting
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Load complete
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 ----------------------------------------
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Creating employee.ism from EMPLOYEE.CNT.LZ4
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Extract starting to /sip/data/employee.cnt
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Record length: 280 Keys 4 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Load starting
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Load complete
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 ----------------------------------------
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Creating inventory.ism from INVENTORY.CNT.LZ4
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Extract starting to /sip/data/inventory.cnt
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Record length: 125 Keys 6 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Load starting
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Load complete
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 ----------------------------------------
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Creating order.ism from ORDER.CNT.LZ4
Oct 12 13:55:45 sipws bash[5891]: 20261012-135545 Extract starting to /sip/data/order.cnt
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Record length: 83 Keys 6 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Load starting
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Load complete
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 ----------------------------------------
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Creating orderline.ism from ORDERLINE.CNT.LZ4
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Extract starting to /sip/data/orderline.cnt
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Record length: 63 Keys 3 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:46 sipws bash[5891]: 20261012-135546 Load starting
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load complete
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 ----------------------------------------
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Creating shipto.ism from SHIPTO.CNT.LZ4
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Extract starting to /sip/data/shipto.cnt
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Record length: 198 Keys 1 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load starting
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load complete
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 ----------------------------------------
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Creating user.ism from USER.CNT.LZ4
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Extract starting to /sip/data/user.cnt
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Record length: 162 Keys 3 Attributes: FIXED,COMPRESS,PAGE=1024,SGRFA,STATIC_RFA,TBYTE
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load starting
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load complete
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 ----------------------------------------
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Creating vendor.ism from VENDOR.CNT.LZ4
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Extract starting to /sip/data/vendor.cnt
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Record length: 270 Keys 7 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load starting
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load complete
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 ----------------------------------------
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Creating warehouse.ism from WAREHOUSE.CNT.LZ4
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Extract starting to /sip/data/warehouse.cnt
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Record length: 158 Keys 2 Attributes: FIXED,COMPRESS,PAGE=4096,STATIC_RFA,TBYTE
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load starting
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Load complete
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 ------------------------------------------------------------
Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Restore of snapshot SNAPSHOT_0.TAR completed. TAR entries 9. Restored 9/9.

As soon as the snapshot is restored, the agent then checks whether there have been changes to the data since the time the snapshot was created. It does this by comparing the CDC topic offset at the time of the snapshot to the current CDC topic offset.

If the current offset is higher than the snapshot offset then the data changed since the beginning of the snapshot, so the agent will reset its pointer in the CDC topic to the snapshot offset and then play forward, applying all changes that occurred after the start of the snapshot to the local replica data files.

In this case there has been no activity since the snapshot was taken, so no catch-up is required, so the data in the local replica data files is in-sync with the system-of-record data.

Oct 12 13:55:49 sipws bash[5891]: 20261012-135549 Replica rehydrated from snapshot 2d35e4a2-5e4b-4e9b-aef0-8cb49bf16813; CDC replay resuming from offset 0
Oct 12 13:55:50 sipws bash[5891]: 20261012-135550 Assigned topic DEFAULT_CDC partition 0 and positioned to offset 0
Oct 12 13:55:50 sipws bash[5891]: 20261012-135550 READY

Later in the workshop you will see a scenario where a snapshot is restored and there has been subsequent activity, so the local catch-up operation will be required.

Oct 08 09:36:45 sipws bash[65167]: 20261008-093645 READY

When you see the agent reporting READY you know that the startup and data restoration is complete, and the agent is now actively monitoring the CDC topic in real time and will apply any changes that occur in the system of record data to the local replica data files.

Problems?

If the Agent is not running, seek assistance from a workshop facilitator.