8 Innovative Ideas for Open Source Beyond Software

Do you agree that Open Source just can’t be applied on software only? Open source philosophy can be applied to any given domain. Over the years open source has proved its potential and there is no brainer that open source model CANNOT work(Off course some dopeheaded VCs still think there is no money in open source! – Who cares!).

But overall, open source is much much beyond just software field. Its more about sharing things. Sharing innovative stuff with you colleagues. I always think about RMS’ words that building a good software is like making a delicious dish. And there are always fellow cooks around to share the recipies.

In another article by Sonali K. Shah and Danese Cooper they say

If you did not share… [others] would not be able to keep up with you. To do or experience something new and fantastic or go another step faster isn’t much fun when you shout ‘Wow! Did you see that!’ and nobody is there to hear you..

I’d like my readers to discuss with me innovative ideas where open source can be used beyond software.

Here are some of my thoughts:

1. Open Source in Space : Space technology can be really more innovative if scientist would use an open source model. The technology could be open so that private groups could fund and adopt

2. Open Source in Retail: I know of a system where the suppliers and retailer would disclose all their profits to their customers. Customers get an advantage of not getting cheated by unknown profit percentage and on the other hand retailers would form a community and sell products.

3. Open source in Mechanics: I am not sure if you guys have heard about MultiMachine: Its an open source all purpose machine which can be built by any semi skilled mechanic. Surprised? Read more about it here: http://opensourcemachine.org/the-multimachine

4. Open source Beer Project: Wow! Sounds fun: Yes, its true… These guys share their beer recipe to the world! More about it at : http://www.opensourcebeerproject.com/

5. Open Source Toys Project: The LLUG (Ladies’ Linux Users Group (in Japan)) started with this concept. They have released paper patterns (they dub this: “source code”) and recipes (“documents”) for four toys by now, and two of them have been distributed as “hand-made kits” (“distributions”). Catch more of it here

6. Open source in Hardware: “Morphy One” is an open source hardware development project in Japan.It claimed to be the first IBM PC/AT compatible- architecture-based palmtop PC produced only by users. The project aims to put out all the data on its development process and specifications under GPL.

7. Open Source in Agriculture: Ever thought of open-source DNA? Could that be the key to agricultural innovation and feeding an ever-growing population? Currently many companies hold patents on crucial agricultural biotechnologies: However, Richard Jefferson believes that those few companies could be using those patents to dominate then destroy an industry. Typically when a company makes an invention, it files for a patent and does not freely share the invention! Could there be something like open source patents?

8. Open source in Politics: How about  a politician disclosing all his moves? Sounds crazy! But there are groups which can form such a model. There is already a concept of Professional Parties where a bunch of professionals form a political party and stand for an election. Who knows if Open source Politics could take this one step further?

Gone are the days of EJB

I hated EJB. I still hate it and will keep hating it. Perhaps its not just hatred but its apathy. Be it EJB 1.x , 2.x or 3.x.  Frameworks come and frameworks go and I don’t feel EJB has survived with respect to the hype it received. I would discuss some major points which get EJB lagging behind:

Too many diversified versions: For those of you who have been with EJB since 1 or 2 would probably agree how it vagabond the versions were. EJB. I once heard Debu Panda from Oracle saying : EJB 1 was like a Queen Honey Bee – Lot of Hype  but did nothing. EJB 2.x came out like an Elephant: Like a mammoth, It could do a lot many stuff but eating away a lot of resources too! Then came EJB 3.x which is like a COW who is much productive and minimal resources.

But the big question is : You still need to write a lot of code! EJB is just not available in the student community. There are quite a few books, but they are industry oriented or rather useless.

Talk of the industry: When I used to take sessions for EJB for young industry guys. I get bombarded with questions on how I could help them use EJB to solve their problems!

Come on guys, EJB is OK but you don’t have to use EJB all the time!!  Its like old days of 90’s when XML was very popuar and people just used XML with every second application! Give me a break!

Fine lets talk about enterprise now as EJB’s E stands for Enterprise. EJB’s model definitely gives you a lot of stuff for the enterprise framework but it surely takes all your resources – Manpower, Infrastructure and mostly contunity. EJB makes you lame. Once you have implemented a lot of things in EJB its quite difficult to move out of it. I hate you EJB for that.  Many of my blog readers are avid Oracle and EJB fans, but sorry guys but EJB has #FAIL ed.

Competition: With EJB 3.x standing still, we have a lot many competing technologies at our hand. EAI frameworks like WebMethods, TIBCO and Kettle (Open Source) gives you far better implementation and reduce overall costs.
Another such framework which has not got the fame it should get is DWR – Direct Web Remoting. TIBCO sponsors this project and its a MUST try out.

The only thing I like about EJB is the persistence API but when things get complex, not many people can do a great job with it. So eventually they still end up doing more stuff.

Overall, I still hate EJB and feel it should reach its end of life!

Marketcetera : Open Source Trading Platform

Most stock trading systems these days are mostly on Solaris, but the trend is changing. Stock broking companies are wanting effective low cost solution. Most companies I know of are moving towards .Net platform or Linux platform. .NET is far cheaper than Solaris solution, but it takes you down over the years with M$ extrating money out of you year after year! So financial companies are also looking at open solutions like Linux.

Adding to this trend, Lets welcome Marketcetera(http://www.marketcetera.org) – Open Source software for automated trading systems. Marketcetera’s platform lets brokers and traders build effective automated trading systems, develop proprietary algorithms, create order management solutions and manage risk faster, easier and at much lower cost than with closed platforms.

Here’s a Quick Overview of the system

Some special features of Marketcetera

  • Process orders through FIX protocol in the exchange. This is good for DMA(Direct Market Access) Clients.
  • Order Routing – You can route your orders. Its an interesting feature which I might discuss it over in another post.
  • Exchange Simulators to test your stuff. You definitely need them, But not quite sure how many exchanges can they simulate? Exchanges in Asia-Pacific markets are like way too different.
  • Order statuses: They kind of flow the orders from Front office to back office, Again this is quite interesting, I will try to blog on it some other day, But yes, those who are familiar with stock broking side will know what Trade Flow is all about.
  • MySQL/Postgres or Oracle can be used as a back-end database. Really cool!
  • It uses the Spring framework with ActiveMQ as the messaging queue – I guess JMS has become heart and soul for all stock brokers these days with Tibco and Webmethods pushing their sales guys hard on this domain. Again a good way to test out the scalability of ActiveMQ with respect to other platforms in the closed source group.
  • It uses various GUI’s – For entering orders it uses Photon, for others,it also uses RoR – Not quite sure if RoR will scale up?! Big question, We’ll see.. I really wonder how much of Web 2.0 are they using!
  • This is not it, there are plenty of features , particularly to hardcore stock broking side like algorithms etc, which I feel you may get an insight at their own site.

I haven’t got enough time to actually put my hands on and try it out , but it seems promising. A few more years of hardcore development and it could be the best! Just needs some experts on the business side which I think their team already has.

But the question is, Will Marketcetera be able to survive when the markets are falling. Offcourse they need a huge investment, I don’t quite know who’s funding them. The one great thing that I like about them is that they are following everything that an open source product needs: Wiki, Mailing list, releases etc. Its really good to see Trading platforms in Open Source. Hat’s off to Marketcetera Inc.

html2pdf using jsp and linux

Have you ever thought of downloading the current viewed page as PDF. Here’a  quick program which can read an HTML and convert it to PDF using JSP or Servlet. I am going to call an external linux command called ‘convert’ which does all the magic for me.

All I need is a linux box with ImageMagick setup along with html2ps and ghostscript. I can get this very very easily from my ubuntu’s apt-get install …

So here’s the small snippet which I wrote after setting up.

try
{
Runtime rt = Runtime.getRuntime() ;
Process p = rt.exec(“convert /usr/local/share/pdfformat1.html /usr/local/share/pdfformat.pdf”  ) ;
p.waitFor();   //<– This waits until the operation is over
System.out.println(p.exitValue());  //<– if the output is 0 it means pdf was generated successfully
}
catch (Exception e){
e.printStackTrace();
}

Now this can be quite a good replacement for iText html to pdf conversion which is quite buggy (I am a huge fan of iText, but somehow this HTML needs to be rendered properly I feel)

After having got the file, read the file and reply to your browser by response.setContentType(“application/pdf”);

Please note, ONLY valid HTML tags are allowed. This method doesn’t support CSS. If someone knows of it, please let me know.This method works perfectly well with valid HTML syntax. So if your output pdf isn’t looking as expected, please check your HTML.

You can use this little utility to check your valid HTML #: sudo apt-get install wdg-html-validator and then run #: validate filename.html
This will list out all the invalid HTML tags.

So for all you report building guys out there, here’s a quick tip: Just output your report as HTML <offcourse with data poluated> and then use this utility to generate a PDF out of it!

Creating sequences in MySQL an alternative to AUTO_INCREMENT

MySQL doesn’t doesn’t have sequences and Auto_Increment does not suffice the need for a text based sequence generator. Or say you want to reset AUTO_INCREMENT’s value?

So here’s a workaround for a generating a sequence in MySQL which can be reset later.

Step 1: Create a Table

CREATE TABLE IF NOT EXISTS `seqgen` (
`seqno` int(4) unsigned NOT NULL,
`application_id` varchar(20) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;

Step 2: Add your application in the sequence
INSERT INTO `seqgen` (`seqno`, `application_id`) VALUES (0, ‘myappname’);
Now when inserting I have set the seqno = 0 , because I want my sequence to start with 0;

Step 3:  You can create a Stored Proc or a Function to get the value and increment it by 1
<Remember to add this Proc/Function through command line with delimiter, else it will not work>

delimiter //
CREATE PROCEDURE seq_gen( OUT nextval INT , IN applicationid text)
BEGIN
select seqno into nextval from seqgen where application_id = applicationid;
update seqgen SET seqno = seqno + 1 where application_id = applicationid;
commit;
END;
//

—OR Function —- If you use function note that you’ll have to commit it by your program, MySQL doesn’t allow implicit or explicit commit in a Stored Function. So better use Procedure..

delimiter //
CREATE FUNCTION f_seq_gen(applicationid text)
RETURNS INT
BEGIN
DECLARE
nextval INT;
select seqno into nextval from seqgen where application_id = applicationid;
update seqgen SET seqno = seqno + 1 where application_id = applicationid;
RETURN nextva;
END;
//

Step 4: Access your sequence by

select f_seq_gen(‘myappname”);

Now you can use many such applications as Sequences in MySQL

WebMethods Document Tracker for Broker analysis

Did you ever have the need to retrieve the documents from the broker without creating another client just for debugging? Try the WebMethods Document Tracker. Its simple and sweet!

It will just take the settings of your broker server and which publisher to subscribe to.

Its quite strong and can handle large documents at ease.

Some common features of the document tracker is as follows:

It can publish a document even with a complex data structure.

It has the ability to export all or selected documents from outgoing or Incoming panel.

It can give you published or delivery reports. Also it can give you statistics of how many documents published in a single go. Or how many currently getting published and delay between two publishes.

Its multi-threaded. This means that publishing and delivery is now in separate threads. This also means you have the ability to browse Incoming documents while doing a pub/sub. Further adding to this, we can even stop the document to be published in the middle. (Highly recommended for debugging/issue analysis at the broker level)

The interface is quite easy to use, although its java Swing with some MFC for windows.

Here are some screen shots for your eyes!

webmethods-document-tracker-pic1

The above screen shows the webmethods document tracker’s main screen. I have highlighted the panel where you can check out the incoming and outgoing documents. The subscribe tab allows you to subscribe to a document.

webmethods-document-tracker-select-broker

This one shows how we can select a broker to monitor. You can even have a secure connection to the broker by providing a certificate file.

WebMethods and TIBCO CUF – Common Utility Framework Services

The Common Utility Framework or CUF is a set of utility services created by any enterprise for their internal usage.

These utility services form the backbone and will provide a set of APIs to the applications which will be written for any enterprise solution.

The major advantage of using a CUF in your enterprise solution is to have a common standard implemented with respect to change of human resource in any organization.

Basic Features of CUF

A CUF should be responsible for everything which is used again and again and a programmer doesn’t like to code for it. All extra work which is most necessary but one doesn’t want to waste time in implementing it should go inside CUF.

CUF services would include:

Auditing (Transaction Logging)
Say you have a long process; you break it down and want to insert logging of the process as checkpoints. You really need some good utility to log all of this. Apache log4j is a good thing, but it needs to get into your CUF so that all other application can access it easily.

One can even log the following:
Complete Message Traceability : Track the entire message
Performance – Time spent in each transaction

Logging (General Logging)
You just need some general debug logging, or any other log.

You may want to put the logs on files or database. It depends on your organization.

There should have varied levels of log. How much data you want to capture?

This CUF service should also be responsible to change the levels of the logs at runtime

Exception Handling
There will always be happy path and sad path, what to do in that case? A common framework to handle common exception scenarios.

You may want to log these errors.

The best practice is to create a generic handler for all common exception.

Error Notification and Escalation
System Errors –Internal Errors, Mis-configuration, Database errors, Timeouts etc.
Business Errors – Component or Application Errors related to resources, processes, static data etc.

One should also keep in mind the following

Error levels – Critical, High, Medium, Low etc.
Alert levels – Whom to escalate, what level?

Escalations procedures – FIX, Root Cause and Prevention mechanism.

Message Repair:
Correct Missing Data – Human is notified, fixing and reprocessing work flow afterward

Reprocessing:
CUF should ideally be responsible or rather say intelligent enough to have reprocess logic
There should be different levels of retry:
For example –Retry from start, retry from last error checkpoint etc.

Medium:
What medium to use for notification?
Emails, Dashboards, alert screens, SMS
This should be configurable as per the interface
It should also be reliable

Initialization and Cleanup

Initial setup on start up
To reserve some memory
To initialize a data feed.
To initialize a pool of objects etc.

Locking mechanism: A set of common locking mechanism for shared services.
To ensure only one instance of the service is running
To ensure concurrency at service call level.

Cleanup of memory
To ensure your service terminates properly
To ensure you release all garbage data and clear unused memory

Custom garbage collectors.
May be a custom garbage collector on the service level.
For example, if a service has been running for more than X hours. Where X is max limit, it should get killed.

Shared Resources:

Database Pooling:
A common database pool of database and adapters which can be used as shared resources.

Middleware:
A message driven middleware or JMS/EMS shared adapters to PUB-SUB messages.

Service Management:
Manage service calls, Thread Management, Interrupt and reset shared object pools.

As a conclusion –A CUF should be responsible to make life easy.

There are hard and fast rules or concrete standards that organizations should follow but the above points should help you getting started to have CUF implemented at your enterprise.