Skip to content

README.md

Latest commit

 

History

History
211 lines (147 loc) · 11.5 KB

README.md

File metadata and controls

211 lines (147 loc) · 11.5 KB

Ingesting data using IBM Streams and running remote applications with IBM Db2 Event Store

Introduction

This document provides information to set up an environment in which you can run the IBM Streams application and remote acess appplications on Event Store. You have the option of either manually setting up the run-time environment or creating a container called eventstore_demo which has everything pre-configured.

The following directories contain step by step instructions on ingesting data using IBM Streams and running external applications using several access methods.

Follow these links to get the specifics for each sample application:

Below are all the commands to run all the tests above (except for IBM Streams) inside the IoT demo container

cd ~/db2eventstore-IoT-Analytics/AdvancedApplications/ScalaApplication
./runscalaExample
cd ~/db2eventstore-IoT-Analytics/AdvancedApplications/PythonApplication
./runpythonExample
cd /root/db2eventstore-IoT-Analytics/AdvancedApplications/ODBCApplication
./bldExampleODBCApp /clidriver
./ExampleODBCApp
cd /root/db2eventstore-IoT-Analytics/AdvancedApplications/JavaApplication
./runjavaExample
cd $HOME/db2eventstore-IoT-Analytics/rest
./install.sh
./run.sh

Environment setup to run sample applications

You have the option of either setting up the environment in a docker container for applications above or picking a specific appliciaton and setting up that specific environment. It is assumed you are running in a RedHat / CentOS environment. If that is not true, use this material as guidance to complete your setup.

Docker container option

Instructions to build a docker container are found here. The container is configured to run all sample applications.

Specific application option

Spark Setup

To install Spark 2.4.8, follow these steps:

  • Go to https://archive.apache.org/dist/spark/spark-2.4.8/
  • Download spark-2.4.8-bin-hadoop2.6.tgz (suggest using "/home/)
  • Download via 
    yum install -y wget
wget https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.6.tgz
  • Where the tar file was downloaded to open the archive using 
    tar -xvf spark-2.4.8-bin-hadoop2.6.tgz
  • In the terminal session the remote application(s) will be run, set the SPARK_HOME variable using the directory where the archived was untarred (e.g., /home/). export SPARK_HOME=/home/<userid>/spark-2.4.8-bin-hadoop2.6

You can also refer to the script used to set up the container if yuu would like to see the details of what was run.

Java Setup

To set up java (for java 1.8) use: 

sudo yum install -y java java-devel

You can also refer to the script used to set up the container f yuu would like to see the details of what was run.

Scala Setup

To set up scala use:

wget http://downloads.typesafe.com/scala/2.11.8/scala-2.11.12.rpm sudo rpm -ihv scala-2.11.12.rpm

You can also refer to the script used to set up the container f yuu would like to see the details of what was run.

SBT Setup

To set up Simple Build Tool (SBT), version 1.6.2, do the following as the root user.

wget -O /tmp/sbt-1.6.2.tgz https://github.com/sbt/sbt/releases/download/v1.6.2/sbt-1.6.2.tgz
tar xzvf /tmp/sbt-1.6.2.tgz -C /usr/share/
ln -s /usr/share/sbt/bin/sbt /usr/bin/sbt

To determine the version in use, run this command:

sbt sbtVersion

You can also refer to the script used to set up the container yuu would like to see the details of what was run.

To install another version of SBT, browse the releases here https://github.com/sbt/sbt/releases/ and make changes to the wget and tar lines with the version you want.

Python Setup

To setup the Python 3.6.13 environment do the following. You will need to have pandas and numpy installed to execute the generate.sh locally(for Centos Linux) .

Below we will download the package from https://www.python.org/ftp/python/3.6.13 and extract the file and perform installations. This will install python in your $HOME directory

cd 
wget https://www.python.org/ftp/python/3.6.13/Python-3.6.13.tgz
tar xzf Python-3.6.13.tgz

cd Python-3.6.13
./configure --enable-optimizations
make altinstall
sudo yum install python-pip
sudo pip install numpy
sudo pip install pandas

Then install pip3 following thes instructions: https://www.liquidweb.com/kb/how-to-install-pip-on-centos-7/, Finally install the PySpark module with this command

pip3 install pyspark

You can also refer to the script used to set up the container yuu would like to see the details of what was run.

ODBC/DB2CLI Setup

If you are using the Docker container, skip this entire ODBC/DB2CLI setup
The IBM Data Server Driver Package is included in the docker container. In the Docker container the <ds_driver_path> is /dsdriver and the <odbc_path> is /clidriver.

To setup the ODBC client environment (for non-Docker container) do the following:

  • Download latest fixpack for version 11.5 GA of the IBM Data Server Driver Package (including the ODBC driver) or IBM Data Server Driver for ODBC and CLI from the site below, selecting your host platform. You are required to have an IBM id in order to download this package. After the file is downloaded you need to copy it to the host or the container from where you intend to run your ODBC application.

    https://www.ibm.com/support/pages/download-initial-version-115-clients-and-drivers
    then go to Download 11.5 fix packs for IBM Data Server Client Packages fix pack download page and get the latest to get the latest, which should be this link
    https://www.ibm.com/support/pages/node/323035

    IBM Data Server Client Packages Fix Packs by version for 11.5

    The download

    IBM Data Server Driver Package

 This package contains drivers and libraries for various programming language environments. It provides support for Java (JDBC and SQLJ), C/C++ (ODBC and CLI), .NET drivers and database drivers for open source languages like PHP and Ruby. The latest driver package from Version 11.5 also includes an interactive client tool called CLPPlus that is capable of executing SQL statements, scripts and can generate custom reports.

    for your operating system, for the demo docker image we use IBM Data Server Driver Package (Linux/x86-64 64 bit) V11.5.6 Fix Pack 0 but as this gets updated these values may change V11.5.6 Fix Pack 0 to something more current

Note: If using the demo container, this "IBM Data Server Driver Package (Linux AMD64 and Intel EM64T)" is already copied into the container and extracted to /dsdriver inside the container.

  • Go the location where the IBM Data Server Driver Package was saved on your host or container and unpack it onto a previously created directory <ds_driver_path>.
tar -xvf v11.5.6_linuxx64_dsdriver.tar.gz -C <ds_driver_path>

old instructions for 11.5 was

tar -xvf ibm_data_server_driver_package_linuxx64_v11.5.tar.gz -C <ds_driver_path>

Since it so confusing to navigate and obtain this package, here are screenshots for the 11.5.6 fix pack 0 download





  • Find the ODBC client package ibm_data_server_driver_for_odbc_cli.tar.gz within <ds_driver_path>. For example, it might be found under dsdriver/odbc_cli_driver/linuxamd64. Then proceed to unpack it onto a previously created directory <odbc_path>.

tar -xvf <ds_driver_path>/odbc_cli_driver/<your platform>/ibm_data_server_driver_for_odbc_cli.tar.gz -C <odbc_path>

  • Then copy the gssplugin libraries from the security32 or security64 directory (depending on your architecture) within the IBM Data Server Driver Package (for example <ds_driver_path>/security64) to the directory where you unpacked your ODBC client (<odbc_path>/clidriver). The gssplugin libraries are used by the ODBC client to connect to Db2 Event Store.

cp -r <ds_driver_path>/security64 <odbc_path>/clidriver

After this, you should be able to find the following file <odbc_path>/clidriver/security64/plugin/IBM/client/IBMIAMauth.so. If the plugin is not found by the ODBC client you will receive back error [IBM][CLI Driver] SQL30082N Security processing failed with reason "17" ("UNSUPPORTED FUNCTION"). SQLSTATE=08001.

Downloading the IBM Db2 Event Store client JAR

To get the client jar go to the following and download the jar:

Downloading the IBM Db2 Event Store JDBC client jar

To get the JDBC client jar go to the following and download the jar:

Downloading the IBM Db2 Event Store Python package

To get the IBM Db2 Event Store python package follow these instructions: 

  • Obtain the Event Store client Python packages here and follow the README:

  • Untar the python.tar into your Python installation directory.

    • For example: tar -xvf python.tar  -C /Library/Frameworks/Python.framework/Version/2.7.5/lib/python2.7/

  • Note that to run python, you will need the following in setup_env.sh

    • export SPARK_HOME=/home/user1/spark-2.0.2-bin-hadoop2.6
    • export ESLIB=<Location of the client jar>/ibm-db2-eventstore-client.jar
    • cp the client jar to $SPARK_HOME/jars
    • export PATH=$PATH:.:$SPARK_HOME/bin
    • export PYTHONPATH=":/usr/local/lib/python2.7/:$SPARK_HOME/python"