A tempest is brewing with Apache Storm – whats going on there?

I’ve lost a bit of faith in the Apache Foundation of late. I understand that a project that is in incubation is still getting its house in order but how long should that take, how much confidence should one put in interim releases and when should you just throw your hands up in the air and say enough is enough, time to find another tool?

My Apache Storm cluster blew up for no apparent reason. I spent days, maybe a week or more debugging esoteric problems that simple test cases for common use cases, should have resolved for an upgrade and also for normal development and testing runs.

After visiting San Francisco in early August, I fired up my Apache Hadoop, Zookeeper, Hbase, Kafka and Storm cluster to crunch some data that I’d collected. It’d been maybe three weeks since the last time. The cluster started but my topology would not run. There was some issue with a Kafka offsets. A few days later, after checking all my maven POMs and dependencies, recompiling, chasing through multiple log files and re-deploying countless times, I came to the realisation that there was nothing actually wrong with my code. Many google searches later, I worked out from one vague little reference, that the Kafka Spout Zookeeper entry was causing a continuous reset of the Kafka Spout to a point that could never be found in the Kafka queue. It was not going to resolve itself. A little while later, I removed the offending Zookeeper entries and the topology started moving. Only to then have other issues to do with timeouts betweens workers and the nimbus supervisor.

This was taking me days to resolve. I kept seeing others having issues with the Apache Storm Incubating 0.9.2 release. Some where similar to mine, others were the normal noob questions posed in user groups. I kept thinking to myself, what is going on there with the Apache Storm Incubating group?

Why aren’t they resolving these common issues being experienced by many? Or at least publishing a fix that doesn’t involve patching the source code and recompiling to create a new distribution. The average developer and smaller shops & startups just don’t want these hassles, nor have the time or skills to be focusing on debugging a release. Like I was eluding earlier, I think people put faith in these releases, as being of a certain quality.

I did see a note, coming through a while back stating that the Apache Storm Incubating committee where looking to introduce Release Candidates before a final release. This is something I’d like to see, but I’d also like to see, a deemphasis of features and bringing in code branches, that have been worked on in isolation eg Yahoos YARN version of Hadoop without the appropriate support. This support should be to make the project, a success as an Apache project. That is, just dropping the code base and saying here it is, isn’t enough.

Clearly there is a lot of interest, in Apache Storm, within the broader Big Data community. There is a lot of good will and people investing significant energy learning and trying to leverage the software. However from my perspective I can’t see the Apache Foundation and the Apache Storm Incubating committee rewarding that faith. The current release is just too unstable and the user community wasting too much time on debugging esoteric problems with no readily available fix.

Releases are taking too long, common issues that make the product usable aren’t being addressed appropriately through the main Apache Storm incubating website http://storm.incubator.apache.org/. Its presently eroding community good will and support to use the product. I get the distinct feeling that the whole process is under resourced and some of the key players, are lost with new features and future releases.

Here in ends my rant. C’mon guys, get what you have working and stabilise it before introducing new features. I want to use Apache Storm, but presently I can’t and no I’m not going to learn Clojure to help.


Technology and my Passion: Nick Hortovanyi on Cycling

Technology and my Passion: Nick Hortovanyi on Cycling

My guest post about cycling and technology on Vinnie Mirchandi’s blog New Florence. New Renaissance – on global technology innovation and impact on how we work, live and play.

We believe

We believe…
Software is changing the world
Data is the lifeblood of change
Developers are the new King makers
More than 2.3 billion Internet users globally connected
One plus billion Internet users in Asia
New Interent users in Asia in 2012: 102,000,000
One plus billion social media users globally
Facebook (900+ Million) Twitter (200+ Million) LinkedIn (150+ Million)
World Wed Web had half a Zetabyte of data in 2009
Zetabytes? 1,000,000,000,000,000,000,000 bytes = 1000 Petabyte, Big Data
In 2013, humans will create 4+ Zetabytes of new data
Stored in massive warehouse sized datacenters
Cloud Computing is changing economics
Re-imagine everything
Old methods aren’t being improved
They are being made obsolete
By Big Data and Machine Learning in the Cloud
Race against the machines?
Join us as we discover more
You ain’t seen anything yet
Toast Technology

Extended and updated 1st January 2013

Big Data – disrupting business?

Maybe it’s the term Big itself, but the mere mention of Big Data causes immense excitement with business executives. This is leading to disruption in business, as they quickly realize that their existing business strategy is inadequate and needs to change.

This has led many to follow past strategies and to be spectators, waiting to see which Use Cases others have wins with.

It’s argued by many, that this approach, whilst successful in the past, will leave many organisations too far behind to catch up. I myself don’t believe, that with this innovation wave, that you are going to see the killer Use Cases shared within an industry. Those that invest and gain competitive advantage won’t share exactly how they do it. They’ll however share the building blocks, as more formal academic research papers. If you are not a technology or Internet company you most likely will be relying on the old guard tech vendors. It will take these vendors time to create solutions – that can be rolled out in an industry. But “where’s the competitive advantage?”, if all your competitors have the same solution.

Why is it so disruptive to business?

We are discovering that the insights gleamed via data and modern machine learning techniques are invalidating previously accepted business approaches and models. In the past, before cheap computing power and data storage was available, it was acceptable to have higher error rates. It was too costly, in the past, to either automate the processes or to collect sufficient data (sampling interval were long or aggregate/derived data was used).

Now it’s not too costly to do so but it’s difficult to assemble a team with the skills to utilize it. It’s even harder, to show that the ingrained learning of your executives, is stopping progress. But that’s what happens, when they learn more about Big Data, it causes business disruption as they realize the old ways are quickly becoming outdated.