underdog-blog » TIBCO

BusinessEvents 4: using Scorecards in a clustered environment

jens — Sat, 23 Oct 2010 18:43:18 +0000

Scorecards have a special purpose in TIBCO BE. There are often used for static (as in Java-style static) values which should be globally accessible through the whole engine. As Java statics, TIBCO describes its main purpose in instance dependent Variables which are only valid in the context of one Processing Unit in an Inference engine.
Here a little citation of the TIBCO documentation on this topic (p.144 – Understanding and Working With Scorecards):

A scorecard is a special type of concept. A scorecard serves as a set of static
variables that is available throughout the project.
…
It is more accurate to say there is one instance of a scorecard per inference agent.
Each inference agent in an application has its own instance of the score card.
Scorecards are not shared between agents.

So you can use Scorecards without any Issues as long as you consider the per Instance rule. Now I have a project where Timeouts are calculated an stored in Scorecards. This solution works perfectly in single instance environment because of the special behavior described above. Now there comes a new Requirement into the project. The Agents should be clustered over 2 Instances to ensure high availability. This really ruins the simplicity of the scorecard approach.

Digging a little deeper into the documentation I came to the conclusion that all the scorecards had to be stored in the cache-server. Knowing this, it came to me mind that the only thing I must accomplish would be, that all inference agents use the same instance of the scorecard.

To achieve the single Instance per Agent approach, TIBCO uses Instance keys for every Processing Unit. So if you could change this instance key, all agents had to us the same instance of the scorecards and the timeout-solution would still work. Searching the documentation the answer was also found really quickly on page 144 (Understanding and Working With Scorecards).
On that page TIBCO described how to set this instance key via CDD.

Any agent that uses scorecards, and also uses Cache Manager, must be assigned a
unique key so that the correct scorecard can be retrieved from the cache. The key
is set in the Processing Unit tab of the CDD.

I also tried this approach on a little test project and it worked right away.

What this means is, you can change the behavior of Scorecards depending on your needs. If you need a per Instance Variable you just have to set different Instance keys for every Agent. On the other hand if you want a globally shared Variable you can achieve this by setting the same value to all instances.

handling a few thousand simultaneous connections in TIBCO BusinessWorks

jens — Tue, 08 Dec 2009 15:22:16 +0000

Scaling with TIBCO BusinessWorks can sometimes be a bit tricky. Recently I began testing some scenarios how to scale a Webservice a bit larger. The first source of information was of course the official documentation and to look at the proposed best practice values for such an engine.
To start small, I tried a HTTP Receiver with a 32bit JVM runtime. I set the heap to the maximum amount possible (something about 1.7gig) and tried how many connections I could handle with that. After a few hundred (300-400) the engine alsways ran into an Out-Of-Memory Exception. From that point the engine was often not recoverable and had to be killed.
After that I tried my luck with an 64bit JVM. Theoretically, with more RAM more connections should be possible, so lets go for it.
I increased the heap size to about 4gig. With that value, the engine actually consumed something about 6gig of memory (I only had 8gig on my test machine). Running the same test as before the connection count increased linearly. That was something I didn’t expect. First of I expeceted the Memory consumption should lower on the amount of connections (because of the more reusable objects) and an increase in CPU load because of the more and more complicated handling of the larger heap on the JVM end.
Despite that I came close to handle about one thousand connections. This seems pretty good but was not enough for what I had in mind. The only possibility I saw at that point was to increase the Memory further and further to get more connections running. A second concern which came to my mind was the thread handling. In a default Tomcat installation ever connection gets its own thread. This does consume a lot of memory but also increases the thread count of the server engine dramatically.
With that in mind I remembered something I read about Tomcat 6 a while ago. For the purpose of handling a lot of simultaneous and enduring Javascript requests Tomcat introduced a new kind of connector engine which uses the Java NIO Framework. To explain this a little, Sun introduced this framework in Java 1.4 to handle IO over a single thread mechanism and use mostly OS provided functions for memory allocation and interaction. So in theory this could be the ideal framework for handling a lot of network IO. Tomcat introduced this feature with the c10k problem specifically in mind. So I began searching around how I could get a similar behvior out of BusinessWorks.
What I didn’t know at that point was, that TIBCO already introduced this feature into BusinessWorks with version 5.7. You can switch the HTTP connector engine with some parameter in the HTTP Connection resource.

HTTP Connection - engine selector

So I changed the connector engine and restartet the test. This time I started small. I set the heap to 512MB and limited the maxProcessor to 500. What I then saw was unexpected. The engine filled up right to the ten thousand requests I send. There occured no Out-Of-Memory at all. What was also interesting was, that the engine held 10k connections despite the maxprocessors was set to 500.
So to conclude the result, the new connector is quite impressive when you need to handle a lot of simultanous connection and have not a lot of memory to spare. On the other hand, when you use it, you loose some of the TIBCO integrated features to limit your load. Further to that, the TIBCO documentation states that due to the single thread arcitecture you increase latency. So as always there is a tradeoff.

One final sidenote. I had some issues with the BusinessWorks 5.7.1 engine so I upgraded it to 5.7.2. Than it ran without a glitch.

TIBCO Designer Panel too small

jens — Tue, 13 Oct 2009 16:52:38 +0000

Recently I ran into some rather trivial problem which isn’t really addressed by the TIBCO Designer. I had a process which wouldn’t fit into the Design Panel. There was just not enough space on the canvas to fit in the actual flow.
After asking around I came to the conclusion that every designer (from different colleges I work with) had a different resolution for the Design canvas. Nobody knew any kind of property where you can set this resolution, so I began searching around.
The designer uses basically 2 Folder for its configs. One is the installation folder with the designer.tra (already explored this one). The other one is the .TIBCO folder in your user home directory. In that folder there exists a file with the name “Designer5.prefs”.
The content of this file consists mainly of position data of the various dialogs. Further to that it includes all the values which you can set through the designer preferences window. Back to the actual topic, I found the following two values:

graph.height.pref=712
graph.width.pref=1515

These values represent the default size which the Designer allocated for its canvas. As you can see the default is pretty small on this one. I also found installations where both values where ten times larger then this. So far I found no performance penalty to this.
The only pattern I found is that newer installation had smaller values as default. Where this value comes from and how it is determined stays unclear to me.

One other thing I found during my research. If the panel is to small for the current process you can drag one activity right next to the border and then start to move it via keyboard (‘Shift + Cursor’). By doing so you drag the icon out of the canvas, but the position will be updated internally. After that you just need to refresh the process view and voila you have expanded the canvas size. But this is just a quick and dirty work-around.

init script for the TIBCO Administrator

jens — Sat, 05 Sep 2009 17:03:06 +0000

I recently ran into the situation that I needed to install a TIBCO BusinessWorks with Administrator onto a RedHat Server. Under Windows the installer provides everthing you need to run your domain as a Service. In Linux this looks different. I have found no init script templates nor did the installer generate me some stubs. So I had to write them myself. So here is what I came up with (I know it isn’t perfect, but it works – suggestions are always welcome).

for the Administrator:

#!/bin/bash

DOMAIN=tibcoesb
USER=esb
TIBCO_HOME=/home/$USER/tibco
PATH=/bin:/usr/bin:/sbin:/usr/sbin
NAME=tibco-admin

start() {
# Start daemons.
echo "Starting Tibco Admin"
/bin/su $USER -c"cd $TIBCO_HOME/administrator/domain/$DOMAIN/bin;./tibcoadmin_$DOMAIN 2>&1 | /usr/bin/logger -t $NAME" &
echo "done"
}

stop() {
# Stop daemons.
echo "Shutting down Tibco Admin"
killall tibcoadmin_$DOMAIN
echo "done"
}

case "$1" in
  start)
	start
	    ;;
  stop)
	stop
    	    ;;
  *)
        echo $"Usage: $0 {start|stop}"
    	exit 2
esac

for the Hawk

#!/bin/bash

DOMAIN=tibcoesb
USER=esb
TIBCO_HOME=/home/$USER/tibco
PATH=/bin:/usr/bin:/sbin:/usr/sbin
NAME=tibco-hawk

start() {
# Start daemons.
echo "Starting Tibco Hawk"
/bin/su $USER -c"cd $TIBCO_HOME/tra/domain/$DOMAIN;./hawkagent_$DOMAIN 2>&1 | /usr/bin/logger -t $NAME" &
echo "done"
}

stop() {
# Stop daemons.
echo "Shutting down Tibco Hawk"
killall hawkagent_$DOMAIN
echo "done"
}

case "$1" in
  start)
	start
	    ;;
  stop)
	stop
    	    ;;
  *)
        echo $"Usage: $0 {start|stop}"
    	exit 2
esac

adding IP-restriction to BusinessWorks processes

jens — Sun, 16 Aug 2009 23:16:07 +0000

Recently I got into the situation that somebody used some interface a way it was not designed for and so created an out-of-memory situation which couldn’t be handled by the engine itself. So now I got the case that one client block the complete service due to invalid requests which he shouldn’t do in the first place.
In the short term there seemed to be only one solution, block the client. So I began to search around how this could be done.

The easiest way to achieve this would be to change the authentication for this service. Unluckily this service is used by a lot of other clients, so I can’t change it out of the blue. Next thing that came to my mind was using the underlying tomcat to do some IP-based blocking. After some searching around I gave that one up. Tibco has a really complex deployment model for BusinessWorks so I wasn’t able to even find the process-specific section for the tomcat engine.
After some searching around in the documentation (actually several hours – hooray to a usable documentation) I found some parameter which allows you to block IPs with some TRA-file parameters. In case you want to look it up yourself it is in the BusinessWorks documentation under ‘Administration’ -> Chapter 8 ‘Custom Engine Properties’ -> ‘Available Custom Engine Properties’ (and another hooray to the excellent formatting – Note to Tibco: HTML has tags, so you can do more with the content then continuous text – even some breaks would enhance the experience).
So back to the point:

bw.plugin.http.server.allowIPAddresses
bw.plugin.http.server.restrictIPAddresses

Both tags are rather self explanatory so you can use right away. Tibco says you can use it with a single IP, a list of IPs (comma-separated) or regular expression. As stated before this highlight of a documentation doesn’t say what reg-exp syntax means. I guess Tibco means they use the Tomcat engine for evaluation of these values an that means they use the Java syntax described here.

writing Unicode characters to Oracle with TIBCO BusinessWorks

jens — Thu, 28 May 2009 12:19:33 +0000

As always, I get the most puzzling mysteries from work requirements. I got the requirement of sending an email with Russian characters through TIBCO BusinessWorks. So far so good. BusinessWorks has full support of Unicode, so it should be not a problem to get this one running. Lucky me the reality looks different.

For this email system a database template system was used and of course the database was an upgraded Oracle 10g, so all the columns (for historic reasons) were Latin-1 and not Unicode.

But a simple Google query even showed a solution for this problem. Oracle introduced, exactly for this case, a new data type (NCHAR – means Unicode character). After changing the table column from CLOB to NCLOB the database should have the ability to store Unicode characters, even if it is running in Latin-1 mode.

First thing I tried, was to insert a valid Russian email template into the database and lucky me, the text which was stored in the database, was still a bunch of invalid Latin-1 characters (character code 0xbf).

After quite some googling I came to this Oracle article. In that article Oracle explains a bit more how to force the JVM to use Unicode characters for the JDBC connection instead of the character set presented from the database. With nothing to lose I tried it in my designer via modifying the designer.tra (I just appended the following parameter).

 -DOracle.jdbc.defaultNChar=true

After that, it worked like a charm. Now it was possible to write Unicode characters to the new created NCLOB columns in the database.

After creating the template and doing a trial run I got curious what would happen if I removed that parameter and tried to read from the Unicode column. To my surprise it worked out-of-box. So you can read Unicode characters from an Oracle database without modifying anything. The indifferences just happen if you want to write via JDBC.

To round this up this bug is not limited to TIBCO BusinessWorks. I also tried it with a tool named DbVisualizer (also Java based) and (what really annoys me, because it is c-based and shouldn’t have this JDBC bug) tora. All showed the same result on writing to that database.

TIBCO Designer with external memory window

jens — Fri, 24 Apr 2009 16:13:10 +0000

During Research for my last blog post I found an interesting feature of the designer. If the designer uses more than one gigabyte of heap memory, the display of the memory usage gets a bit fuzzy.

Obviously somebody at TIBCO forgot to round the value, so it would fit into this small section of the status bar.
Keeping that in mind I found a start parameter which allows you to start the designer with an external window which only displays the current memory load.

As you can see this is much more useful then the default.
To get this you just have to start the designer with the following command:

./designer -memory

I wonder how much more (undocumented) functionality is hidden in the designer.

improve TIBCO Designer tester performance under linux

jens — Tue, 21 Apr 2009 22:20:25 +0000

I’m using the TIBCO designer for quite a while now. Before using it in a debian environment I developed all TIBCO related stuff in Windows XP. Now with the switch to linux there came quite a shift in user experience. One thing that really annoyed me was the slow performance of the designer debugger.
So I started some measurements with a simple test process. The test process creates a simple list of all files (2000 items) in one folder and then iterates over every entry. Here is what I got:

Windows XP: 194sec
Windows XP (minimized): 6sec
Gnome: 470sec
Gnome (minimized): 4sec

All this data was gathered with the same default installation of a TIBCO designer 5.6 with the default java runtime. The configs were all left untouched. So now I tried to find something to improve that behavior. I first concentrated on how to influence the jvm.
After a bit of research in the TIBCO direction a found the following value in the tra-file which allows the user to pass parameters directly to the jvm:

java.extended.properties

With that information I tried several parameters suggested by google. After a few tries I came to this one:

sun.java2d.pmoffscreen=false

What Sun says about it isn’t really clear to me but it helps drastically to improve performance. Back to my original test I came up with the following timings:

Gnome (pmoffscreen=false): 75sec
Gnome (pmoffscreen=false): 5sec

As you can see it actually surpasses the Windows installation. That was a result I didn’t actually expect. Till now I found no drawback to this solution.
Just for the sake of completeness here my full config line of the designer.tra

java.extended.properties=-Xrs -Xmx3072M -Xms1536M -XX:+AggressiveOpts -XX:-UseParallelGC -XX:-UseConcMarkSweepGC -XX:MaxPermSize=512M -XX:+UseFastAccessorMethods -Xverify:none -Dsun.java2d.pmoffscreen=false

installing TIBCO TRA 5.6 on a debian 64bit

jens — Mon, 23 Feb 2009 22:59:40 +0000

Recently I got a hardware upgrade so I could finally switch to a 64-bit environment. To fully use that machine I wanted to install TIBCO in 64-bit mode. After starting the installation I got this message:

TIBINS202527: Error: ERROR: You are running a 64-bit product installer on a 32-bit system.
This is not supported.

The Problem was I was running a 64-bit OS with a 64-bit Kernel:

 uname -a
Linux client1 2.6.28.5 #1 SMP PREEMPT Tue Feb 17 17:42:33 CET 2009 x86_64 GNU/Linux

So I tried to find the problem. First I needed some more output what the installer is actually doing. So I used the logging option to get all the debug output.

 ./TRA.5.6.0-suite_linux24gl23_x86.bin -console -is:log output

So when you look closer to this log you can see that the command which is running the installer looks like this:

Executing launch script command: “/tmp/isjI8lFYy/bin/java” -cp “”:”TRA.5.6.0-suite_linux24gl23_x86.jar”:”TRA.5.6.0-simple_linux24gl23_x86.jar”:”tibrv.8.1.1-simple_linux24gl23_x86.jar”:”jre.1.5.0-simple_linux24gl23_x86_64.jar”:”Designer.5.6.0-simple_linux24gl23_x86_64.jar”:”tpcl.5.6.0-simple_linux24gl23_x86.jar”:”hawk.4.8.1-simple_linux24gl23_x86_64.jar”:”/tmp/isjA5jEsB/TRA.5.6.0-suite_linux24gl23_x86.jar”:”" -Dtemp.dir=”/tmp” -Dis.jvm.home=”/tmp/isjI8lFYy” -Dis.jvm.temp=”1″ -Dis.media.home=”/tmp/isjA5jEsB/TRA.5.6.0-suite_linux24gl23_x86.jar” -Dis.launcher.file=”/home/jens/tmp/TIB_tra-suite_5.6.0_linux24gl23_x86_64/./TRA.5.6.0-suite_linux24gl23_x86.bin” -Dis.jvm.file=”/tmp/isjI8lFYy/jvm” -Dis.external.home=”/home/jens/tmp/TIB_tra-suite_5.6.0_linux24gl23_x86_64/.” -Xms20m -Xmx128m run -home TRA.5.6.0-suite_linux24gl23_x86.jar “-console”

Now that I had the command which is starting the installer I began to trace what this process is doing. To do a strace properly you just have to prepend ‘strace’ and than redirect the error output to a file (cause it is quiet a lot). So after doing this I found something interesting in the log.

[pid 32651] execve(“/bin/uname”, [0xffffffffdf47dc88, "-p"], [/* 1757 vars */]) = 0

As you can see it runs the command ‘uname -p’. This command returns ‘unknown’ for a default debian system. You also have this problem if you are running a self compiled kernel from kernel.org. As for the TIBCO supported systems (SUSE ans redhat) they return something different. After trying the same command on a openSUSE I found that ‘x86_64′ should be the correct string. After I bit of trial and error I found out that the result of this command is written to a file name ‘kernelbits_jens.txt’ in the temp directory.

So here the simple solution to the problem.
You just need to create the arch file manually and make it read-only so the installer can’t overwrite it. Here the command:

echo 'x86_64' > kernelbits_`whoami`.txt

Now the installer worked absolutely fine for me.
I already notified the TIBCO support about the problem. As for now there will be no fix. But I hope they will correct this behavior for future installer.

Tibco EMS with database backend (postgresql)

jens — Sun, 16 Nov 2008 18:48:11 +0000

I recently tried to build a JMS Server with database backend. The chosen product was the TIBCO EMS Server. The Server brings its own database support over hibernate.

Unfortunately TIBCO supports only Oracle,Mysql and DB2 by default. Lucky me, I needed an installation for Postgres but this shouldn’t be a big deal, because hibernate supports postgres as well. You just have to modify the hibernate config.

First you should install the EMS server and hibernate (version provided by Tibco). After that let’s start configuring.

To have everthing in the database you need 3 databases (3 separate EMS stores). For me I used the following 3 stores:

emsmeta -> for metadata content
emsnf -> for nonfailsafe data
emsf -> for failsafe data

I used the same account for all 3 databases (just easier for testing purposes).

Setup the stores in the stores.conf in the EMS folder:

[$sys.meta]
type=dbstore
dbstore_driver_url=jdbc:postgresql://localhost/emsmeta
dbstore_driver_username=ems
dbstore_driver_password=ems

[$sys.nonfailsafe]
type=dbstore
dbstore_driver_url=jdbc:postgresql://localhost/emsnf
dbstore_driver_username=ems
dbstore_driver_password=ems

[$sys.failsafe]
type=dbstore
dbstore_driver_url=jdbc:postgresql://localhost/emsf
dbstore_driver_username=ems
dbstore_driver_password=ems

After that you have to change you tibemsd.conf file.The following line is a sample what you should add. Additionally you should add tha path to your jdbc driver (in this sample the last parameter – change it to your config). You also have to add the path to your JVM (here it is the debian lenny default).

dbstore_classpath       = ../../../components/eclipse/plugins/com.tibco.tpcl.org.hibernate_3.2.5.001/hibernate3.jar:../../../components/eclipse/plugins/com.tibco.tpcl.org.com.mchange.c3p0_0.9.1.001/c3p0-0.9.1.jar:antlr-2.7.6.jar:asm-attrs.jar:asm.jar:cglib-2.1.3.jar:commons-collections-2.1.1.jar:commons-logging-1.0.4.jar:dom4j-1.6.1.jar:ehcache-1.2.3.jar:jta.jar:/usr/local/bin/oracledrivers/lib/ojdbc5.jar:/home/jens/tibco/tpcl/5.6/jdbc/postgresql-8.3-603.jdbc3.jar

## db section
#dbstore_driver_name     = oracle.jdbc.driver.OracleDriver
#dbstore_driver_dialect  = org.hibernate.dialect.Oracle10gDialect
dbstore_driver_name     = org.postgresql.Driver
dbstore_driver_dialect  = org.hibernate.dialect.PostgreSQLDialect
jre_library             = /usr/lib/jvm/java-6-sun/jre/lib/i386/server/libjvm.so

As you can see changing databases is easy. You just have to change the driver_name and driver_dialect of hibernate.

After that you have the basic configuration ready. Next thing to do is to initialize the dabases. For that task Tibco provides a tool which generates the proper sql-statements.

You can run the following command in your shell and get the sql as output.

java -jar tibemsd_util.jar -tibemsdconf tibemsd.conf -createall

For some reason the create sql-statements had no ‘;’ at the end of every command. So you can’t just pipe the output to psql. You have to do it old school.

After that you can start the EMS Server and it stores all data to the selected database.