I am still a novice when it comes to the technical underpinnings of databases and Hadoop. So, I thought it might be useful if I just asked if my thought on the future of databases is correct.
Basically, “Does Hadoop signal the end of the database as we know it?’
Here’s where this question comes from:
I work for Netezza, who make blazingly fast data warehouse appliances. At the heart of the appliance is a Postgres database. But due to the appliance architecture (and I think the fast speed) you don’t have to do the usual things you have to do to make databases work, such as tuning, indexing, and so forth (indeed, we have a long list of “no”s that set database folks crazy, as in “How can you not do that?”).
That got me thinking. Our appliance has changed the need for coddling databases. Indeed, weren’t databases created to make it easier for (what back then were) slow computers to handle large amounts of data, and all the coddling is to compensate for weak hardware? Would we need databases if it didn’t matter how the data was structured, as long as we had a fast search and processing of the data?
Segue to Hadoop
Lately, at work we’ve been taking about Hadoop, hearing folks actually NOT wanting to have a structured database. And, we see folks with large amounts of data with Hadoop, just throwing more processing power at the data when needed.
Following that thread, I started wondering if the evolution of tools like Hadoop might make structure databases obsolete*, that it really doesn’t matter how the data is structured, just so long as we can find it. And the processing issues are obviated by just throwing more processing nodes at it.**
So, teach me:
Where am I wrong in this thought thread? Will data always need to be structured somehow for computing purposes? How much of the structured data world can Hadoop gobble up (though the unstructured data world must be larger than the structured data world, right?)?
What do you think?
*Of course, just like folks are still using VAX, databases will really never disappear. When a technology is displaced, it usually doesn’t disappear, just gets relegated to a different niche.
**Do you still keep things in folders? I only do when I don’t have a good search tool. On my Mac, I use Spotlight to find and open anything, rather than searching through folders. Indeed, everything usually goes into one folder. Unless I need to separate something for follow up on the desktop (so, OK, folder doe not go away altogether). Nonetheless, search has replaced most of what I would use folders for.
