I’ve lost a bit of faith in the Apache Foundation of late. I understand that a project that is in incubation is still getting its house in order but how long should that take, how much confidence should one put in interim releases and when should you just throw your hands up in the air and say enough is enough, time to find another tool?
My Apache Storm cluster blew up for no apparent reason. I spent days, maybe a week or more debugging esoteric problems that simple test cases for common use cases, should have resolved for an upgrade and also for normal development and testing runs.
After visiting San Francisco in early August, I fired up my Apache Hadoop, Zookeeper, Hbase, Kafka and Storm cluster to crunch some data that I’d collected. It’d been maybe three weeks since the last time. The cluster started but my topology would not run. There was some issue with a Kafka offsets. A few days later, after checking all my maven POMs and dependencies, recompiling, chasing through multiple log files and re-deploying countless times, I came to the realisation that there was nothing actually wrong with my code. Many google searches later, I worked out from one vague little reference, that the Kafka Spout Zookeeper entry was causing a continuous reset of the Kafka Spout to a point that could never be found in the Kafka queue. It was not going to resolve itself. A little while later, I removed the offending Zookeeper entries and the topology started moving. Only to then have other issues to do with timeouts betweens workers and the nimbus supervisor.
This was taking me days to resolve. I kept seeing others having issues with the Apache Storm Incubating 0.9.2 release. Some where similar to mine, others were the normal noob questions posed in user groups. I kept thinking to myself, what is going on there with the Apache Storm Incubating group?
Why aren’t they resolving these common issues being experienced by many? Or at least publishing a fix that doesn’t involve patching the source code and recompiling to create a new distribution. The average developer and smaller shops & startups just don’t want these hassles, nor have the time or skills to be focusing on debugging a release. Like I was eluding earlier, I think people put faith in these releases, as being of a certain quality.
I did see a note, coming through a while back stating that the Apache Storm Incubating committee where looking to introduce Release Candidates before a final release. This is something I’d like to see, but I’d also like to see, a deemphasis of features and bringing in code branches, that have been worked on in isolation eg Yahoos YARN version of Hadoop without the appropriate support. This support should be to make the project, a success as an Apache project. That is, just dropping the code base and saying here it is, isn’t enough.
Clearly there is a lot of interest, in Apache Storm, within the broader Big Data community. There is a lot of good will and people investing significant energy learning and trying to leverage the software. However from my perspective I can’t see the Apache Foundation and the Apache Storm Incubating committee rewarding that faith. The current release is just too unstable and the user community wasting too much time on debugging esoteric problems with no readily available fix.
Releases are taking too long, common issues that make the product usable aren’t being addressed appropriately through the main Apache Storm incubating website http://storm.incubator.apache.org/. Its presently eroding community good will and support to use the product. I get the distinct feeling that the whole process is under resourced and some of the key players, are lost with new features and future releases.
Here in ends my rant. C’mon guys, get what you have working and stabilise it before introducing new features. I want to use Apache Storm, but presently I can’t and no I’m not going to learn Clojure to help.
Nick Hortovanyi's blog 