Monthly Archives: January 2009

Webmethods to TIBCO Migration

I’ve been getting a lot of hits and requests to post something about Webmethod to Tibco migration. As discussed in many of my previous posts, Tibco is highly marketing is products and particularly asking organizations to migrate from WebMethods!  You should see the way Tibco sales guys talk! They are technically convincing!

SO lets try to discuss if its really a good decision to migrate from Webmethods to TIBCO

Reason

You must have a reason to migrate. Don’t migrate just because someone told you. Or you are getting good Tibco resource available. There are plenty of resources for Webmethods too.

You must migrate because you find something missing in WebMethods. Or something which takes a lot of time in Webmethods than Tibco.

You must migrate if supporting an application in Webmethods is more difficult than webmethods.

Problems & Strategy

So lets say you have found your big reason to actually migrate from Webmethods to TIBCO. So what are the next steps:

1. Study the architecture. There is a huge change from Hub and Spoke architecture to Bus architecture. Webmethods runs an integration server. You can run each packages / services in the single server. But Tibco is Bus Architecture. Each package/service will run as a separate BWEngine. That means you’ll end up having a lot of system level processes on the server.

2. There will be no centralised management. In webmethods, you can share a lot of common services and call them from inside. But in Tibco you need to include the utility services as part of your EAR. Please note, while running things from Tibco designer, this is not the case.

This results in usage of extra memory. Lets say you have a lot of common business services, Its absolutely impossible to include each one of them as a library in your final EAR. Hence people tend to deploy these libraries as a separate BWEngine. Again a bigger problem is to do interprocess communication. For this EMS is used. A typical example is the TIBCO CUF(Common Utility framework ). All this becomes too complex for something simple.

Tibco Administrator is again single point of management for all TIBCO BWEngines and its again a good one but not quite complex as WebMethods’ Admin panel.

What TIBCO doesn’t give you: The JMS doesn’t even have a web based EMS Queue viewer… (I plan to write it someday! Or Tibco Guys should pay me for that! !)

Tibco Admion also doesn’t give you statistics about how much CPU is consumed or how much memory is being used.

However there is a workaround for that by using Tibco HAWK. Again another product that you may have to buy.

Supporting

While supporting Tibco applications, one cannot debug it unless the code is taken from the repository or installed on your system.
You will have to manage your Tibco code through your own repository. Applying patches in Tibco is also not very matured as Webmethods.

My Choice

I would still be reluctant to migrate to TIBCO. Not because I know Webmethods More than I know TIBCO, but because I feel webmethods is quite a good product. Tibco is more of a hype(I’m sure Tibco guys will hate me for saying this). But facts are facts. I get more stuff working and in an easy way in webmethods than TIBCO!

Reader’s Choice:

I would request the readers to post their own comments.

Webmethods:How to Kill a Running Service from the Integration Server

Many of you have wondered how to kill a WebMethods service while its running on the Integration Server.

Its become a Pet Interview Question for many but the interviewer also expects one of the following facts which is not the most recommended method to kill a running service.

Here are some facts:

1. If you are running the service on debug mode, the service will get killed if you kill your developer.
2. If you are running the service on RUN mode, the service runs on the Integration Server and will not get killed.
3. If your client has activated the service but terminated the session, the service will still run on the IS.
4. If your webmethod service is hung, You will try to Reload the entire package. But this will NOT help.
5. You will also try to unload the package but this will NOT help too.

What you need to do is on a Thread Level. The idea is quite simple: Just get the list of all the current running threads and kill the service which you need.

I will try to explain this in a step by step manner. Developers please do try it out on your dev boxes as well.

Step 1:
Create a Flow service like this:

wm1

Step 2:
Lets create a Flow service which will get ALL running threads. This should take NO input but lets output a String containing all running threads on that Integration Server.

w2

Step 3:

Insert the following Java Code in your WebMethods Flow Service that you just created. The code is commented well enough.

//—BEGIN CODE—–
// Author: Hussain Fakhruddin (hussulinux@gmail.com)
// Date: 20090128
//Get the Current Thread
Thread currentThread = Thread.currentThread();
//Get all Thread Groups.
ThreadGroup rootThreadGroup = currentThread.getThreadGroup();
//Traverse to the Top Thread Level
while (rootThreadGroup.getParent() != null) {
rootThreadGroup  = rootThreadGroup.getParent();
}
//Getting all Threads
Thread [] threads = new Thread[1000];
int threadCount = rootThreadGroup.enumerate(threads, true);
String threadList = “”;
// Now traverse through all the Threads and use the ServerThread API from webmethods to get all the thread detail.
int i=0;
for (; i<threadCount ; ++i){
if (threads[i] instanceof com.wm.app.b2b.server.ServerThread) {
//Casting a raw thread into ServerThread type
com.wm.app.b2b.server.ServerThread serverThread = (com.wm.app.b2b.server.ServerThread) threads[i];
//Getting the service Name for which the thread belongs to
if (serverThread.getState().getService() != null){
threadList = threadList + serverThread.getState().toString() ;
java.util.Stack threadStack = (java.util.Stack) serverThread.getState().getCallStack();
for (Iterator iter=threadStack.iterator(); iter.hasNext();) {
Object threadObj = iter.next();
threadList = threadList + “\n ” + threadObj.getClass().getName();
threadList = threadList + “, ” + threadObj.toString();
}
}
}
}
IDataCursor cursor = pipeline.getCursor();
cursor.insertAfter(“threads”, threadList);
cursor.destroy();

//—–END OF CODE—–

Step 4: Now lets create another flow service “killThread” with the following inputs and outputs

wm3

Step 5: Now copy paste the following code. We’re going to traverse through the entire list and call the .interrupt() method to kill the process. Note: you need to provide FULL path of the service which you want to kill. You can get the full path from the webmethods admin panel (under package management).

//—BEGIN CODE—–
// Author: Hussain Fakhruddin(hussulinux@gmail.com)
// Date: 20090128
IDataCursor cursor = pipeline.getCursor ();
if (cursor.first (“serviceName”)) {
String serviceName = (String) cursor.getValue ();
Thread current = Thread.currentThread ();
ThreadGroup root = current.getThreadGroup ();
while (root.getParent () != null) {
root = root.getParent ();
}
Thread [] threads = new Thread [1000];
int count = root.enumerate (threads, true);
String sb = “Not Found”;
StringBuffer killedList = new StringBuffer ();
for (int i=0; i<count; i++) {
if (threads[i] instanceof com.wm.app.b2b.server.ServerThread) {
com.wm.app.b2b.server.ServerThread serverThread = (com.wm.app.b2b.server.ServerThread) threads[i];
if (serverThread.getState ().getService () != null) {
java.util.Stack stack = (java.util.Stack) serverThread.getState ().getCallStack ();
for (Iterator iter=stack.iterator (); iter.hasNext ();) {
Object obj = iter.next ();
String name = obj.toString ();
if (name.trim().equals (serviceName)) {
threads[i].interrupt ();
sb = “Interrupted”;
killedList.append (stack.toString ()).append (“\n”);
}
}
}
}
}
cursor.insertAfter (“status”, sb);
cursor.insertAfter (“killedList”, killedList.toString ());
cursor.destroy ();
}
//—–END OF CODE—–

That’s it folks! Save and run it.. Come back to me if you face problems..

8 Innovative Ideas for Open Source Beyond Software

Do you agree that Open Source just can’t be applied on software only? Open source philosophy can be applied to any given domain. Over the years open source has proved its potential and there is no brainer that open source model CANNOT work(Off course some dopeheaded VCs still think there is no money in open source! – Who cares!).

But overall, open source is much much beyond just software field. Its more about sharing things. Sharing innovative stuff with you colleagues. I always think about RMS’ words that building a good software is like making a delicious dish. And there are always fellow cooks around to share the recipies.

In another article by Sonali K. Shah and Danese Cooper they say

If you did not share… [others] would not be able to keep up with you. To do or experience something new and fantastic or go another step faster isn’t much fun when you shout ‘Wow! Did you see that!’ and nobody is there to hear you..

I’d like my readers to discuss with me innovative ideas where open source can be used beyond software.

Here are some of my thoughts:

1. Open Source in Space : Space technology can be really more innovative if scientist would use an open source model. The technology could be open so that private groups could fund and adopt

2. Open Source in Retail: I know of a system where the suppliers and retailer would disclose all their profits to their customers. Customers get an advantage of not getting cheated by unknown profit percentage and on the other hand retailers would form a community and sell products.

3. Open source in Mechanics: I am not sure if you guys have heard about MultiMachine: Its an open source all purpose machine which can be built by any semi skilled mechanic. Surprised? Read more about it here: http://opensourcemachine.org/the-multimachine

4. Open source Beer Project: Wow! Sounds fun: Yes, its true… These guys share their beer recipe to the world! More about it at : http://www.opensourcebeerproject.com/

5. Open Source Toys Project: The LLUG (Ladies’ Linux Users Group (in Japan)) started with this concept. They have released paper patterns (they dub this: “source code”) and recipes (“documents”) for four toys by now, and two of them have been distributed as “hand-made kits” (“distributions”). Catch more of it here

6. Open source in Hardware: “Morphy One” is an open source hardware development project in Japan.It claimed to be the first IBM PC/AT compatible- architecture-based palmtop PC produced only by users. The project aims to put out all the data on its development process and specifications under GPL.

7. Open Source in Agriculture: Ever thought of open-source DNA? Could that be the key to agricultural innovation and feeding an ever-growing population? Currently many companies hold patents on crucial agricultural biotechnologies: However, Richard Jefferson believes that those few companies could be using those patents to dominate then destroy an industry. Typically when a company makes an invention, it files for a patent and does not freely share the invention! Could there be something like open source patents?

8. Open source in Politics: How about  a politician disclosing all his moves? Sounds crazy! But there are groups which can form such a model. There is already a concept of Professional Parties where a bunch of professionals form a political party and stand for an election. Who knows if Open source Politics could take this one step further?

Gone are the days of EJB

I hated EJB. I still hate it and will keep hating it. Perhaps its not just hatred but its apathy. Be it EJB 1.x , 2.x or 3.x.  Frameworks come and frameworks go and I don’t feel EJB has survived with respect to the hype it received. I would discuss some major points which get EJB lagging behind:

Too many diversified versions: For those of you who have been with EJB since 1 or 2 would probably agree how it vagabond the versions were. EJB. I once heard Debu Panda from Oracle saying : EJB 1 was like a Queen Honey Bee – Lot of Hype  but did nothing. EJB 2.x came out like an Elephant: Like a mammoth, It could do a lot many stuff but eating away a lot of resources too! Then came EJB 3.x which is like a COW who is much productive and minimal resources.

But the big question is : You still need to write a lot of code! EJB is just not available in the student community. There are quite a few books, but they are industry oriented or rather useless.

Talk of the industry: When I used to take sessions for EJB for young industry guys. I get bombarded with questions on how I could help them use EJB to solve their problems!

Come on guys, EJB is OK but you don’t have to use EJB all the time!!  Its like old days of 90’s when XML was very popuar and people just used XML with every second application! Give me a break!

Fine lets talk about enterprise now as EJB’s E stands for Enterprise. EJB’s model definitely gives you a lot of stuff for the enterprise framework but it surely takes all your resources – Manpower, Infrastructure and mostly contunity. EJB makes you lame. Once you have implemented a lot of things in EJB its quite difficult to move out of it. I hate you EJB for that.  Many of my blog readers are avid Oracle and EJB fans, but sorry guys but EJB has #FAIL ed.

Competition: With EJB 3.x standing still, we have a lot many competing technologies at our hand. EAI frameworks like WebMethods, TIBCO and Kettle (Open Source) gives you far better implementation and reduce overall costs.
Another such framework which has not got the fame it should get is DWR – Direct Web Remoting. TIBCO sponsors this project and its a MUST try out.

The only thing I like about EJB is the persistence API but when things get complex, not many people can do a great job with it. So eventually they still end up doing more stuff.

Overall, I still hate EJB and feel it should reach its end of life!

Marketcetera : Open Source Trading Platform

Most stock trading systems these days are mostly on Solaris, but the trend is changing. Stock broking companies are wanting effective low cost solution. Most companies I know of are moving towards .Net platform or Linux platform. .NET is far cheaper than Solaris solution, but it takes you down over the years with M$ extrating money out of you year after year! So financial companies are also looking at open solutions like Linux.

Adding to this trend, Lets welcome Marketcetera(http://www.marketcetera.org) – Open Source software for automated trading systems. Marketcetera’s platform lets brokers and traders build effective automated trading systems, develop proprietary algorithms, create order management solutions and manage risk faster, easier and at much lower cost than with closed platforms.

Here’s a Quick Overview of the system

Some special features of Marketcetera

  • Process orders through FIX protocol in the exchange. This is good for DMA(Direct Market Access) Clients.
  • Order Routing – You can route your orders. Its an interesting feature which I might discuss it over in another post.
  • Exchange Simulators to test your stuff. You definitely need them, But not quite sure how many exchanges can they simulate? Exchanges in Asia-Pacific markets are like way too different.
  • Order statuses: They kind of flow the orders from Front office to back office, Again this is quite interesting, I will try to blog on it some other day, But yes, those who are familiar with stock broking side will know what Trade Flow is all about.
  • MySQL/Postgres or Oracle can be used as a back-end database. Really cool!
  • It uses the Spring framework with ActiveMQ as the messaging queue – I guess JMS has become heart and soul for all stock brokers these days with Tibco and Webmethods pushing their sales guys hard on this domain. Again a good way to test out the scalability of ActiveMQ with respect to other platforms in the closed source group.
  • It uses various GUI’s – For entering orders it uses Photon, for others,it also uses RoR – Not quite sure if RoR will scale up?! Big question, We’ll see.. I really wonder how much of Web 2.0 are they using!
  • This is not it, there are plenty of features , particularly to hardcore stock broking side like algorithms etc, which I feel you may get an insight at their own site.

I haven’t got enough time to actually put my hands on and try it out , but it seems promising. A few more years of hardcore development and it could be the best! Just needs some experts on the business side which I think their team already has.

But the question is, Will Marketcetera be able to survive when the markets are falling. Offcourse they need a huge investment, I don’t quite know who’s funding them. The one great thing that I like about them is that they are following everything that an open source product needs: Wiki, Mailing list, releases etc. Its really good to see Trading platforms in Open Source. Hat’s off to Marketcetera Inc.

html2pdf using jsp and linux

Have you ever thought of downloading the current viewed page as PDF. Here’a  quick program which can read an HTML and convert it to PDF using JSP or Servlet. I am going to call an external linux command called ‘convert’ which does all the magic for me.

All I need is a linux box with ImageMagick setup along with html2ps and ghostscript. I can get this very very easily from my ubuntu’s apt-get install …

So here’s the small snippet which I wrote after setting up.

try
{
Runtime rt = Runtime.getRuntime() ;
Process p = rt.exec(“convert /usr/local/share/pdfformat1.html /usr/local/share/pdfformat.pdf”  ) ;
p.waitFor();   //<– This waits until the operation is over
System.out.println(p.exitValue());  //<– if the output is 0 it means pdf was generated successfully
}
catch (Exception e){
e.printStackTrace();
}

Now this can be quite a good replacement for iText html to pdf conversion which is quite buggy (I am a huge fan of iText, but somehow this HTML needs to be rendered properly I feel)

After having got the file, read the file and reply to your browser by response.setContentType(“application/pdf”);

Please note, ONLY valid HTML tags are allowed. This method doesn’t support CSS. If someone knows of it, please let me know.This method works perfectly well with valid HTML syntax. So if your output pdf isn’t looking as expected, please check your HTML.

You can use this little utility to check your valid HTML #: sudo apt-get install wdg-html-validator and then run #: validate filename.html
This will list out all the invalid HTML tags.

So for all you report building guys out there, here’s a quick tip: Just output your report as HTML <offcourse with data poluated> and then use this utility to generate a PDF out of it!

Creating sequences in MySQL an alternative to AUTO_INCREMENT

MySQL doesn’t doesn’t have sequences and Auto_Increment does not suffice the need for a text based sequence generator. Or say you want to reset AUTO_INCREMENT’s value?

So here’s a workaround for a generating a sequence in MySQL which can be reset later.

Step 1: Create a Table

CREATE TABLE IF NOT EXISTS `seqgen` (
`seqno` int(4) unsigned NOT NULL,
`application_id` varchar(20) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Step 2: Add your application in the sequence
INSERT INTO `seqgen` (`seqno`, `application_id`) VALUES (0, ‘myappname’);
Now when inserting I have set the seqno = 0 , because I want my sequence to start with 0;

Step 3:  You can create a Stored Proc or a Function to get the value and increment it by 1
<Remember to add this Proc/Function through command line with delimiter, else it will not work>

delimiter //
CREATE PROCEDURE seq_gen( OUT nextval INT , IN applicationid text)
BEGIN
select seqno into nextval from seqgen where application_id = applicationid;
update seqgen SET seqno = seqno + 1 where application_id = applicationid;
commit;
END;
//

—OR Function —- If you use function note that you’ll have to commit it by your program, MySQL doesn’t allow implicit or explicit commit in a Stored Function. So better use Procedure..

delimiter //
CREATE FUNCTION f_seq_gen(applicationid text)
RETURNS INT
BEGIN
DECLARE
nextval INT;
select seqno into nextval from seqgen where application_id = applicationid;
update seqgen SET seqno = seqno + 1 where application_id = applicationid;
RETURN nextva;
END;
//

Step 4: Access your sequence by

select f_seq_gen(‘myappname”);

Now you can use many such applications as Sequences in MySQL

WebMethods Document Tracker for Broker analysis

Did you ever have the need to retrieve the documents from the broker without creating another client just for debugging? Try the WebMethods Document Tracker. Its simple and sweet!

It will just take the settings of your broker server and which publisher to subscribe to.

Its quite strong and can handle large documents at ease.

Some common features of the document tracker is as follows:

It can publish a document even with a complex data structure.

It has the ability to export all or selected documents from outgoing or Incoming panel.

It can give you published or delivery reports. Also it can give you statistics of how many documents published in a single go. Or how many currently getting published and delay between two publishes.

Its multi-threaded. This means that publishing and delivery is now in separate threads. This also means you have the ability to browse Incoming documents while doing a pub/sub. Further adding to this, we can even stop the document to be published in the middle. (Highly recommended for debugging/issue analysis at the broker level)

The interface is quite easy to use, although its java Swing with some MFC for windows.

Here are some screen shots for your eyes!

webmethods-document-tracker-pic1

The above screen shows the webmethods document tracker’s main screen. I have highlighted the panel where you can check out the incoming and outgoing documents. The subscribe tab allows you to subscribe to a document.

webmethods-document-tracker-select-broker

This one shows how we can select a broker to monitor. You can even have a secure connection to the broker by providing a certificate file.