Earlier this month at DevOpsDays here in Austin the Dell Crowbar crew hosted a session and gave a demo. If you’re not familiar with it, Crowbar is an open source software framework written at Dell. I grabbed some time with Crowbar architect Rob Hirschfeld and got him to recap how far we’ve come in its less than a year and where he sees us going over the next year.
Extra-credit reading
2 Comments | 
Uncategorized | Tagged: cloud, crowbar, developers, DevOps, open source, Rob Hirschfeld | 
Permalink
Posted by Barton George
Last week DevOpsDays was held here in Austin. It sold out in about day after it was announced and had a big waiting list. The two-day event, which was held at National Instruments (who did an awesome job as host), featured talks and panels in the mornings and “open space” discussions in the afternoons.
The panel on the first day, moderated by John Willis, was entitled: Provisioning Panel – Meet Juju, Crowbar, Puppet, Chef, Pallet + discussion. After the panel I caught up with each of the members for a follow-up chat. Here they are:
Juju – Mark Mimms of Canonical
Crowbar – Rob Booth of Zenoss
Puppet – Dan Bode of Puppet Labs
Chef – Matt Ray of Opscode
Pallet – Antoni Batchelli of Pallet Ops
Stay tuned for more DevOpsDays goodness in the days to come!
Extra-credit reading
Pau for now…
1 Comment | Uncategorized | Tagged: Antoni Batchelli, austin, chef, crowbar, dan bode, devopsdays, juju, mark mimms, matt ray, national instruments, oppscode, palletops, puppet, rob booth, zenoss | Permalink
Posted by Barton George
As I mentioned in my previous entry, the code for the Hadoop barclamps is now available at our github repo.
To help you through the process, Crowbar lead architect Rob Hirschfeld has put together the two videos below. The first, Crowbar Build (on cloud server), shows you how to use a cloud server to create a Crowbar ISO using the standard build process. The second, Advanced Crowbar Build (local) shows how to build a Crowbar v1.2 ISO using advanced techniques on a local desktop using a virtual machine.
Crowbar Build (on cloud server)
Advanced Crowbar Build (local)
Pau for now…
1 Comment | Uncategorized | Tagged: barclamps, build, centos, cloud, crowbar, hadoop, ISO, open source, rackspace, ubuntu | Permalink
Posted by Barton George
Earlier this month we announced that Dell would be open sourcing the Crowbar “barclamps” for Hadoop. Well today is the day and the code is now available at our github repo.
Whats a Crowbar barclamp?
If you haven’t heard of project Crowbar it’s a software framework developed at Dell that started out as an installation tool for OpenStack. As the project grew beyond installation to include monitoring capabilities, network discovery, performance data gathering etc., the developers behind it, Rob Hirschfeld and Greg Althaus, decided to rewrite it to allow modules to plug into the basic Crowbar functionality. These modules or “barclamps” allow the framework to be used by a variety of projects. Besides the OpenStack and Hadoop barclamps written by Dell, VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.
To help you get your bearings
As I mentioned in the opening paragraph, the code for the Hadoop barclamp is now available. To help you get started, below are a couple of videos that Rob put together. The first walks you through how to install Crowbar and the second one explains how to use Crowbar to deploy Hadoop.
Extra-credit reading
Pau for now…
6 Comments | Uncategorized | Tagged: barclamps, Greg Althaus, hadoop, open source, open stack, Rob Hirschfeld | Permalink
Posted by Barton George
In the previous entry I mentioned that we have developed and will be opensourcing “barclamps” (modules that sit on top of Crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop. All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters.
If you would like to get involved, check out this 1 min video from Rob Hirschfeld talking about how:
Look for the code on the Crowbar GitHub repo by the last week of November.
Extra-credit reading:
Pau for now…
2 Comments | Uncategorized | Tagged: crowbar, github, hadoop, openstack, Rob Hirschfeld | Permalink
Posted by Barton George
Rob Hirschfeld, aka “Commander Crowbar,” recently posted a blog entry looking back at how 
Crowbar came to be, how its grown and where he hopes it will go from here.
What’s a Crowbar?
If you’re not familiar with Crowbar, its an open source software framework that began life as an installation tool to speed installation of OpenStack on Dell hardware. The project incorporates the Opscode Chef Server tool and was originally created here at Dell by Rob and Greg Althaus. Just four short months ago at OSCON 2011 the project took a big step forward when, along with the announcement of our OpenStack solution, we announced that we were opensourcing it.
DevOps-ilicous
As Rob points out in his blog, as we were delivering Crowbar as an installer a collective light bulb went off and we realized the role that Chef and tools like it play in a larger movement taking place in many Web shops today: the movement of DevOps.
The DevOps approach to deployment builds up systems in a layered model rather than using packaged images…Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.
On beyond installation and OpenStack
As the team began working more with Crowbar, it occurred to them that its use could be expanded in two ways: it could be used to do more than installation and it could be expanded to work with projects beyond OpenStack.
As for functionality, Crowbar now not only installs and configures but once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
The first project beyond OpenStack that we used Crowbar on was Hadoop. In order to expand Crowbar’s usage we created the concept of “barclamps” which are in essence modules that sit on top of the basic Crowbar functionality. After we created the Hadoop barclamp, others picked up the charge and VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.
It takes a community
Crowbar development has recently been moved out into the open. As Rob explains,
This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders. Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github.
So what are you waiting for? Join our mailing list, download the code or ISO, create a barclamp, make your voice heard. Who’s next?
Extra-credit reading:
Pau for now…
Leave a Comment » | Uncategorized | Tagged: chef, cloud foundry, crowbar, DevOps, dreamhost, Greg Althaus, hadoop, openstack, opscode, Rob Hirschfeld | Permalink
Posted by Barton George
In case you’re not familiar with Cloud Foundry, it’s an open source Platform as a Service 
project initiated at VMware. More specifically it provides a platform for building, deploying, and running cloud apps using Spring for Java developers, Rails and Sinatra for Ruby developers, Node.js and other JVM frameworks including Grails.
The project began two years ago when VMware’s CEO Paul Maritz recruited Derek Collison and Mark Lucovsky out of Google and set them to working on Cloud Foundry. Collison and Lucovsky, who built and maintained Google’s API services, were brought into leverage their experience of working with hugely scaled out architectures.
The Cloud Foundry project has only been public for a matter of months and one question that I’m sure has popped into your mind is what if I want to pilot Cloud Foundry in my own environment, won’t installation and configuration be a total pain?
Enter the Crowbar
Crowbar is an open source software framework developed at Dell to speed up the installation 
and configuration of open source cloud software onto bare metal systems. By automating the process, Crowbar can reduce the time needed for installation from days to hours.
The software is modular in design so while the basic functionality is in Crowbar itself, “barclamps” sit on top of it to allow it work with a variety of projects. The first use for crowbar was for OpenStack and the barclamp for that has been donated to the community. Next came The Dell | Cloudera solution for Apache Hadoop and, just recently, Dreamhost announced that they currently working on a Ceph barclamp. And now…
Two great tastes that taste great together
Today’s big news is that VMware is working with Dell to release and maintain a Crowbar barclamp that, in conjunction with Crowbar, will install and configure Cloud Foundry. This capability, which will include multi-node configs over time, will allow organizations and service providers the ability to quickly and easily get pilots of Cloud Foundry up and running.
Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
If you’d like to try out Crowbar for yourself, check out: https://github.com/DellCloudEdge
Press added after initial posting
Extra-credit reading
Pau for now…
15 Comments | Uncategorized | Tagged: cloud foundry, crowbar, DevOps, google, paas, vmware | Permalink
Posted by Barton George
Today at OSCON we are announcing the availability of the Dell OpenStack Cloud Solution along with the open sourcing of the code behind our Crowbar software framework.
The Solution
Dell has been a part of the OpenStack community since day one a little over a year ago and today’s news represents the first available cloud solution based on the OpenStack platform. This Infrastructure-as-a-service solution includes a reference architecture based on Dell PowerEdge C servers, OpenStack open source software, the Dell-developed Crowbar software and services from Dell and Rackspace Cloud Builders.
Crowbar, keeping things short and sweet
Bringing up a cloud can be no mean feat, as a result a couple of our guys began working on a software framework that could be used to quickly (typically before coffee break!) bring up a multi-node OpenStack cloud on bare metal. That framework became Crowbar. What Crowbar does is manage the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in a matter of hours (or even minutes) instead of days.
Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
Code to the Community
As mentioned above, today Dell has released Crowbar to the community as open source code (you can get access to it the project’s GitHub site). The idea is allow users to build functionality to address their specific system needs. Additionally we are working with the community to submit Crowbar as a core project in the OpenStack initiative.
Included in the Crowbar code contribution is the barclamp list, UI and remote API’s, automated testing scripts, build scripts, switch discovery, open source Chef server. We are currently working with our legal team to determine how to release the BIOS and RAID which leverage third party components. In the meantime since it is free (as in beer) software, although Dell cannot distribute it, users can directly go the vendors and download the components for free to get that functionality.
More Crowbar detail
For those who want some more detail, here are some bullets I’ve grabbed from Rob “Mr. Crowbar” Hirschfeld’s blog:
Important notes:
On the roadmap:
Extra-credit reading
Pau for now…
12 Comments | Uncategorized | Tagged: chef, crowbar, dell openstack cloud solution, iaas, open source, openstack, oscon, poweredge c, rackspace cloud builders | Permalink
Posted by Barton George
Last week on Day two of Structure the morning sessions ended with an interesting discussion moderated by James Urquhart. The session was entitled “DevOps – Reinventing the Developers Role in the Cloud Age” and featured Luke Kanies – CEO, Puppet Labs and Jesse Robbins – Co-Founder and CEO, Opscode.
After lunch I ran into Jesse and got him to sit down with me and provide some more insight into DevOps as well as explain what Opscode was doing with project Crowbar.
Some of the ground Jesse covers
Extra-credit reading:
Leave a Comment » | Uncategorized | Tagged: chef, crowbar, DevOps, iaas, jesse robbins, openstack, opscode, structure | Permalink
Posted by Barton George
We’ve been watching the ARM market develop over the past few years as these highly efficient chips that have been driving tablets and cell phones have been finding their way more and more into hyperscale servers. Well watch no more, today were are sallying forth. Why now? Because some of our biggest customers have told us that they felt the time is now to start working with these low powered, highly efficient chips for their servers.
HW + SW = Solution
Today we announced that we will be shipping the new Dell “Copper” ARM servers via a seed unit program to select hyperscale customers worldwide. But a server does not an ecosystem make so we are doing what we can to help partners and developers get started building out applications for the platform. Given that two of the key areas where the extreme efficiencies of ARM play particularly well are Web front-ends and Hadoop environments, we have “ARMed” key partners like Canonical and Cloudera with units.
Early days
At this point it is still early days in the world of ARM servers so we designed Copper specifically for developers and customers to create code and test performance, not for production. To help developers get started we have struck a partnership between the Dell Solutions Centers and Texas Advanced Computing Center (TACC) to provide devs with remotely accessible clusters to develop and collaborate on. And speaking of developers, Dells own devs are working to deliver an ARM-based version of our open source infrastructure management software, Crowbar.
Speeds and Feeds
And in case your wondering about the specs of the hardware:
Stay tuned for more…
Extra-credit reading
Pau for now…
Leave a Comment » | Uncategorized | Tagged: ARM, canonical, cloudera, copper, developers, ecosystem, TACC | Permalink
Posted by Barton George
Last summer at OSCON Dell announced the availability of our OpenStack solution in the US and 
Canada. Today at World Hosting Days in Rust Germany we are now announcing that our OpenStack-Powered Cloud Solution is available in Europe and Asia.
If you’re not familiar with it, OpenStack is an open source cloud project built on a foundation of code initially donated by NASA and Rackspace. The project kicked off a little over a year and a half ago here in Austin and it has gained amazing traction since then.
Dell’s offering
Dell’s OpenStack cloud offering is an open source, on premise cloud solution based on the OpenStack platform running on Ubuntu. Its composed of:
Crowbar software framework
To give a little more background on the Crowbar software framework, its an open source project developed initially at Dell and you can grab it off github. The framework, which is under the Apache 2.0 license, manages the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in hours, as opposed to days.
Once the initial deployment is complete, you can use Crowbar to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting. Beyond Dell, companies like VMware, Dreamhost and Zenoss have built “barclamps” that allow them to utilize Crowbar’s modular design. Additionally, customers who buy the Dell OpenStack-Powered Cloud Solution get training, deployment, and support on Crowbar.
So as of today, customers in the UK, Germany and China can purchase the Dell OpenStack-Powered Cloud Solution. As customer demand grows in other regions we will be adding more countries so stay tuned. If the first 18 mos of the project are any indication of whats the pace is like to come, we are all going to be in for a lot more excitement.
For more info, email: OpenStack@Dell.com
Extra-credit reading
Pau for now…
6 Comments | Uncategorized | Tagged: asia, crowbar, europe, openstack, poweredgec, ubuntu | Permalink
Posted by Barton George
At our sales kickoff in Vegas, Rob Hirschfeld chose a unique vehicle to succinctly convey our Big Data story here at Dell. Check out the video below to hear one of our chief software architects for our Big Data and OpenStack solutions explain, in less than 90 seconds, what we are up to in the space and the value it brings customers.
Extra credit reading
Pau for now…
1 Comment | Uncategorized | Tagged: big data, hadoop, openstack, Rob Hirschfeld | Permalink
Posted by Barton George
With O’Reilly’s big data conference Strata coming up in just a couple of weeks, I thought I might 
as well get around to finally writing up my notes from Hadoop World . The event, which was put on by Cloudera, was held last November 8-9 in New York city. There were over 1,400 attendees from 580 companies and 27 countries with two thirds of the audience being technical.
Growing beyond geek fest
The event itself has picked up significant momentum over the last three years going from 500 attendees, to 900 the second year, to over 1400 this past year. The tone has gone from geek-fest to an event focused also on business problems e.g. one of the keynotes was by Larry Feinsmith, managing director of the office of the CIO at JP Morgan Chase. Besides Dell, other large companies like HP, Oracle and Cisco also participated.
As a platinum sponsor, Dell had both a booth and a technical presentation. At the event we announced that we would be open sourcing the Crowbar barclamp for Hadoop and at out booth we showed off the Dell | Hadoop Big Data Solution which is based on Cloudera Enterprise.
Cutting’s observations
Doug Cutting, the father of Hadoop, Cloudera employee and chairman of the Apache software foundation, gave a much anticipated keynote. Here are some of the key things I caught:
Interviews from the event
To get more of the flavor of the event here is a series of interviews I conducted at the show, plus one where I got the camera turned on me:
Extra-credit reading
Blogs regarding Dell’s crowbar announcement
Hadoop Glossary
2 Comments | Uncategorized | Tagged: apache foundation, barclamps, big data, crowbar, doug cutting, event, hadoop world, oreilly, strata | Permalink
Posted by Barton George
This is the last in my three-part Web Glossary series. As I previously explained, in compiling this I pulled information from various and sundry sources across the Web including Wikipedia, community and company web sites and the brain of Cote.
The idea behind the glossary is to help our teams get a better understand of the wild and wacky world of the Web and Web developers as we move forward with our Web|Tech vertical. I figured I might as also share it with a few friends.
Today’s focus, having worked our way down from the top, is the infrastructure tier (with a short catch-all bucket at the end , “Misc.”)
Infrastructure
General Terms
Projects/Entities
Misc
Extra-credit reading
Pau for now…
1 Comment | Uncategorized | Permalink
Posted by Barton George
This afternoon Matt Ray, Technical Evangelist for Opscode, stopped by Dell’s Round Rock HQ to brief a gaggle of folks on what they are up to. Cote arranged the visit as well as one last month with Puppet labs, which I unfortunately wasn’t able to make.
After Matt, with some help from teammates on the phone, briefed the Dell gang I grabbed some time with him to get the 5 minute Reader’s Digest version. Here is the result.
Some of the ground Matt covers:
Extra-credit reading
Pau for now…
1 Comment | Uncategorized | Tagged: chef, cloud, crowbar, DevOps, matt ray, openstack, ops code | Permalink
Posted by Barton George
Besides interviewing a bunch of people at Hadoop World, I also got a chance to sit on the other side of the camera. On the first day of the conference I got a slot on SiliconANGLE’s the Cube and was interviewed by Dave Vellante, co-founder of Wikibon and John Furrier, founder of SiliconANGLE.
-> Check out the video here.
Some of the ground we cover
Extra-credit reading
Pau for now…
Leave a Comment » | Uncategorized | Tagged: barton george, crowbar, Dave Vellante, DevOps, hadoop, hadoop world, John Furrier, open source, open stack, SiliconANGLE, the cube | Permalink
Posted by Barton George
It wouldn’t be surprising if you were surprised to learn that Dell is developing software. To say that this is an area we haven’t been known for in the past would be an understatement. While we may not pose a direct threat to Microsoft any time soon, we have been coding in a few focused areas. One of those areas is cloud installation and management and is represented by our project Crowbar. While Crowbar began life simply as a way to install Openstack on Dell hardware, it has expanded from there.
Today’s news is that we have developed and will be opensourcing “barclamps” (modules that sit on top of crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop. All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters. But don’t take my word for it. Take a listen to Crowbar’s architect Rob Hirschfeld as he explains Crowbar and today’s announcement:
Look for the code on Crowbar GitHub repo by the last week of November. If you want to get involved, learn how.
Extra-credit reading:
Pau for now…
3 Comments | Uncategorized | Tagged: barclamps, chef, crowbar, hadoop, open source, openstack, opscode, Rob Hirschfeld | Permalink
Posted by Barton George
If you’re planning on attending the OpenStack Design summit and conference next week in Beantown you’ll have to check us out. I’m bummed that I will be missing the summit for the first time, I have a big internal presentation next week, but the rest of the Dell OpenStack crew will be there in force. Dell is a sponsor at the event and we will have a keynote, speaking sessions and demos.
What have we got in the works?
Besides checking out Crowbar and our Openstack solution which we launched back at OSCON we will have a whisper suite where we will be showing our latest and greatest stuff that is currently in the works. If you’d like to see what we have up our sleeve, email us at OpenStack@Dell.com and we can schedule a time slot for you to come and see for yourself.
Updated: For more details what we’ll be doing at the summit check out Rob’s blog
Extra-credit reading
Pau for now…
1 Comment | Uncategorized | Tagged: cloud, conference, crowbar, Design Summit, nasa, openstack, whisper suite | Permalink
Posted by Barton George
Dell has been working for the last four plus years outfitting the biggest of the big web superstars like Facebook and Microsoft Azure with infrastructure. More recently we have been layering software such as Hadoop, OpenStack and crowbar on top of that infrastructure. This has not gone unnoticed by web pub GigaOm:
Want to become the next Amazon Web Services or Facebook? Dell could have sold you the hardware all along, but now it has the software to make those servers and storage systems really hum.
They also made the following observation:
Because [Dell] doesn’t have a legacy [software] business to defend, it can blaze a completely new trail that has its trailhead where Oracle, IBM and HP leave off.
Letting customers focus on what matters most
Its a pretty exciting time to be at Dell as we continue to move up the stack outfitting web players big and small. The idea is to get these players established and growing in an agile and elastic way so they can concentrate on serving customers rather than building out their underpinning software and systems.
Stay tuned for more!
Extra-credit reading
Pau for now…
Leave a Comment » | Uncategorized | Tagged: crowbar, facebook, gigaom, hadoop, Microsoft Azure, openstack, web | Permalink
Posted by Barton George
Data continues to grow at an exponential rate and no place is this more obvious than in the Web space. Not only is the amount exploding but so is the form data’s taking whether that’s transactional, documents, IT/OT, images, audio, text, video etc. Additionally much of this new data is unstructured/ semi-structured which traditional relational databases were not built to deal with.
Enter Hadoop, an Apache open source project which, when combined with Map Reduce allows the analysis of entire data sets, rather than sample sizes, of structured and unstructured data types. Hadoop lets you chomp thru mountains of data faster and get to insights that drive business advantage quicker. It can provide near “real-time” data analytics for click-stream data, location data, logs, rich data, marketing analytics, image processing, social media association, text processing etc. More specifically, Hadoop is particularly suited for applications such as:
The Dell | Cloudera solution
Although Hadoop is a very powerful tool, it can be a bit daunting to implement and use. This fact wasn’t lost on the founders of Cloudera who set up the company to make Hadoop easier to used by packaging it and offering support. Dell has joined with this Hadoop pioneer to provide the industry’s first complete Hadoop Solution (aptly named “the Dell | Cloudera solution for Apache Hadoop”).
The solution is comprised of Cloudera’s distribution of Hadoop, running on optimized Dell PowerEdge C2100 servers with Dell PowerConnect 6248 switch, delivered with joint service and support. Dell offers two flavors of this big data solution: Cloudera’s distribution with the free download of Hadoop software, and Cloudera’s enterprise version of Hadoop that comes with a charge.
It comes with its own “crowbar” and DIY option
The Dell | Cloudera solution for Apache Hadoop also comes with Crowbar, the recently open-sourced Dell-developed software, which provides the necessary tools and automation to manage the complete lifecycle of Hadoop environments. Crowbar manages the Hadoop deployment from the initial server boot to the configuration of the main Hadoop components allowing users to complete bare metal deployment of multi-node Hadoop environments in a matter of hours, as opposed to days. Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect a complete data analytics solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.
The solution also comes with a reference architecture and deployment guide, so you can assemble it yourself, or Dell can build and deploy the solution for you, including rack and stack, delivery and implementation.
Some of the coverage (added Aug 12)
Extra-credit reading
Pau for now…
9 Comments | Uncategorized | Tagged: Apache, big data, cloudera, crowbar, hadoop, map reduce, powerconnect 6248 switch, poweredge C2100 | Permalink
Posted by Barton George
You have searched the Barton's Blog blog archives for ‘Crowbar’. If you are unable to find anything in these search results, you can try one of these links.
Barton’s Blog & Podcasts
