As I mentioned in my previous entry, the code for the Hadoop barclamps is now available at our github repo.
To help you through the process, Crowbar lead architect Rob Hirschfeld has put together the two videos below. The first, Crowbar Build (on cloud server), shows you how to use a cloud server to create a Crowbar ISO using the standard build process. The second, Advanced Crowbar Build (local) shows how to build a Crowbar v1.2 ISO using advanced techniques on a local desktop using a virtual machine.
Earlier this month we announced that Dell would be open sourcing the Crowbar “barclamps” for Hadoop. Well today is the day and the code is now available at our github repo.
Whats a Crowbar barclamp?
If you haven’t heard of project Crowbar it’s a software framework developed at Dell that started out as an installation tool for OpenStack. As the project grew beyond installation to include monitoring capabilities, network discovery, performance data gathering etc., the developers behind it, Rob Hirschfeld and Greg Althaus, decided to rewrite it to allow modules to plug into the basic Crowbar functionality. These modules or “barclamps” allow the framework to be used by a variety of projects. Besides the OpenStack and Hadoop barclamps written by Dell, VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.
To help you get your bearings
As I mentioned in the opening paragraph, the code for the Hadoop barclamp is now available. To help you get started, below are a couple of videos that Rob put together. The first walks you through how to install Crowbar and the second one explains how to use Crowbar to deploy Hadoop.
I’m getting near the end of the interviews that I did while at Hadoop World earlier this month, just one more after this (with Splunk’s CTO and co-founder).
Today’s entry features a talk I had with Nosh Petigara, director of product strategy at 10gen, the company behind MongoDB.
Some of the ground that Nosh covers
Who is 10gen and what is MongoDB
(0:29) How does Nosh define NoSQL
(1:20) What use cases is Mongo best at
(2:14) Some examples of customers using Mongo (foursquare, Disney and MTV) and what they’re using it for
(3:08) How Mongo and Hadoop work together
(4:03) Whats in Mongo’s future that Nosh is excited about
Extra-credit reading
Mongo Conference: MongoSV (Dec 9 in Silicon valley)
Todd Papaioannou has been in Big Data for a while. He built the original engineering team at Greenplum, worked at Teradata for 5 years and mostly recently, before joining Battery Ventures as an Entrepreneur in Residence, served as Yahoo’s Chief Cloud architect.
I grabbed some time with Mr. P to learn what it means to be an EIR and what he’s seeing in the industry from his vantage point.
Some of the ground Todd covers
Todd’s background
(0:45) What is an Entrepreneur in Residence and how did Todd become one
(2:45) What trends is he seeing in the space and how does he feel the market’s evolving
(4:00) What are his big take aways from this year’s Hadoop World
But wait, there’s more!
Stay tuned for more interviews from last week’s Hadoop world. On tap are:
I’m always interested in what’s happening at Canonical and with Ubuntu. Last week at Hadoop World I ran into a couple of folks from the company (coincidentally both named Mark but neither Mr. Shuttleworth). Mark Mims from the server team was willing to chat so I grabbed some time with him to learn about what he was doing at Hadoop World and what in the heck is this “charming” Juju?
Some of the ground Mark covers
Making the next version of Ubuntu server better for Hadoop and big data
(0:34) What are “charms” and what do they have to do with service orchestration
(2:05) Charm school and learning to write Juju charms
(2:54) Where does “Orchestra” fit in and how can it be used to spin up OpenStack
(3:40) What’s next for Juju
But wait, there’s more!
Stay tuned for more interviews from last week’s Hadoop world. On tap are:
One thing Hadoop isn’t great at right out of the box is data analytics, that’s where a company like Karmasphere comes in. Karmasphere provides business intelligence software that data analysts can use to use to mine the data that Hadoop sucks up.
Last week at Hadoop World I grabbed some time with Karamsphere’s Chairman and co-founder, Martin Hall to learn more about where he and his company play in the wild world of big data.
Some of the ground Martin covers
Where does Karmasphere play in the big data stack, how is it used and by whom
(0:38) Where did the idea for developing Karmasphere come from
(1:58) What is the Karmasphere “secret sauce”
(2:18) What are the main industries and use cases where their offerings are used
(3:40) What can we look forward to in future releases
But wait, there’s more!
Stay tuned for more interviews from last week’s Hadoop world. On tap are: Mark Mims of Canonical, Todd Papaioannou from Battery Ventures, John Gray of Facebook, Erik Swan of Splunk and Nosh Petigara of 10gen/MongoDB.
The next in my series of video interviews from Hadoop World is with Mark Azad who covers technical solutions for Couchbase. If you’re not familiar with Couchbase it’s a NoSQL database provider and the company was formed when, earlier this year, CouchOne and Membase merged.
Here’s what Mark had to say.
Some of the ground Mark covers
What is Couchbase and what is NoSQL
How Couchbase works with Hadoop
What its product line up looks like and his new combined offering coming next year
Some of Couchbase’s customers and how Zynga uses them
What excites Mark the most up the upcoming year in Big Data
Yesterday, Hadoop World 2011 wrapped here in New York. During the event I was able to catch up with a bunch of folks representing a wide variety of members of the ecosystem. On the first day I caught up with Ed Dumbill of O’Reilly Media who writes about big data for O’Reilly Radar and also is the GM for O’Reilly’s big data conference, Strata.
Here’s what Ed had to say.
Some of the ground Ed covers
What is Strata and what does it cover
How will this years conference differ from last
Which customer types are making the best use of Hadoop, will Strata verticalize going forward
What is Ed looking forward to most in the upcoming Strata.
A few hours ago the third annual Hadoop World conference wrapped up here in New York city. It has been a packed couple of days with keynotes, sessions and exhibits from all sorts of folks within the greater big data ecosystem.
I caught up with master of ceremonies and Cloudera CEO Mike Olson to get his thoughts on the event and predictions for next year.
Some of the ground Mike covers:
How this year’s event compares to the first two and how its grown (it ain’t Mikey Rooney anymore)
(2:06) Key trends and customers at the event
(4:02) Mike’s thoughts on the Dell/Cloudera partnership
(5:35) Looking forward to Hadoop world 2012 and where to go next
Stay tuned
If you’re interested in seeing more interviews from Hadoop World 2011 be sure to check back. I have eight other vlogs that I will be posting in the upcoming days with folks from Mongo DB, O’Reilly Media, Facebook, Couchbase, Karmasphere, Splunk, Ubuntu and Battery Ventures.
Yesterday Hadoop World kicked off here in New York city. As part of the opening keynotes, Ping Li of Accel Partners got on stage and announced that they are opening a $100 million dollar fund focusing on big data. If you’re not familiar with Accel, they are the venture capital firm that have invested in such hot companies as Facebook, Cloudera, Couchbase, Groupon and Fusion IO.
I grabbed some time with Ping at the end of the sessions yesterday to learn more about their fund:
Some of the ground Ping covers:
What areas within the data world the fund will focus on.
Who are some of the current players within their portfolio that fall into the big data space.
What trends Ping’s seeing within the field of Big Data.
How to engage with Accel and why it would make sense to work with them.
In the previous entry I mentioned that we have developed and will be opensourcing “barclamps” (modules that sit on top of Crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop. All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters.
If you would like to get involved, check out this 1 min video from Rob Hirschfeld talking about how:
It wouldn’t be surprising if you were surprised to learn that Dell is developing software. To say that this is an area we haven’t been known for in the past would be an understatement. While we may not pose a direct threat to Microsoft any time soon, we have been coding in a few focused areas. One of those areas is cloud installation and management and is represented by our project Crowbar. While Crowbar began life simply as a way to install Openstack on Dell hardware, it has expanded from there.
Today’s news is that we have developed and will be opensourcing “barclamps” (modules that sit on top of crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop. All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters. But don’t take my word for it. Take a listen to Crowbar’s architect Rob Hirschfeld as he explains Crowbar and today’s announcement:
One of the interviews I did at Dell World was a video with IT in Canada. I did the video with Paul Cooper, Dell’s country manager for Canada.
In the first half of the video I talk about how Dell got into the cloud and where we play in the space. In the second half Paul talks about the roll the telcos will play in the delivery of cloud services in Canada as well as issues around privacy and data sovereignty.
Check it out.
From the article itself, here’s a great summary of our cloud participation and shows how we have built, bought and partnered along the way:
Dell’s excursion into cloud began with organic development of server and data centre capability in specialized systems to meet the needs of large cloud providers (Facebook, Microsoft Azure and Bing), progressed through modification of these systems for marketing to the “next 1,000”, and shifted to partnership with software makers such as Joyent to develop complete cloud solutions, and with companies such as VMware for the creation of a full service public cloud offering.
Supporting acquisitions along the way include companies with specific capabilities such as SecureWorks, which was purchased to address web security concerns that continue to dog broader cloud adoption, and BOOMI, a specialist in cloud integration, which enables Dell to better service customers who adopt a hybrid cloud approach to sourcing compute resources.
Posted by Barton George 
