For years, it had been drummed into me, to use a relational database system (RDBMS). It was something that was just accepted, there wasn’t any point arguing against it, as it appeared to just work. Data, the lifeblood of any system, could even be recovered in the event of some unfortunate hardware failure, if it was installed properly.
I worked on a few projects, at the end of the .com era, where we ran into difficulties with using relational databases. The issues primarily related to the volumes of data and types of transactions(large number of small inserts and updates) being used. On one project, when the vendor was approached with the issue, they acknowledged it, and said that it was an inherent issue with their product and that they had no plans to address it (so much for paid support, huh?).
Yes, my time honoured faith in the RDBMS had been eroded. But what were the alternatives? It was commercial suicide to not put a proposal forward that didn’t use a RDBMS as the main data store.
A little while ago, I came across an interesting paper "The End of an Architecural Era (It’s Time for a Complete Rewrite)"(PDF here).Now this paper, had the thought processes going. In it, they present reasons and strong evidence that the "one size fits all" commercial RDBMS paradigm is coming to an end. A key point that stood out for me is that they are 25 year old legacy code lines, written originally for 1970s architected hardware. The hardware has been retired, so should that code line in my humble opinion as well. Its well worth the time to read the paper.
"What are the alternatives?", I hear you still asking. Well I came across an excellent blog post by Richard Jones, "Anti-RDBMS: A list of distributed key-value stores".
I’ve been experimenting on a prototype, very successfully with Project Voldemort instead of a RDBMS. So next time, you are looking for a persistance store, don’t assume you are best served with a RDBMS. It may no longer be the case!