DevOpsDays: Crowbar, where its been and where its going

April 25, 2012

Earlier this month at DevOpsDays here in Austin the Dell Crowbar crew hosted a session and gave a demo.  If you’re not familiar with it, Crowbar is an open source software framework written at Dell.  I grabbed some time with Crowbar architect Rob Hirschfeld and got him to recap how far we’ve come in its less than a year and where he sees us going over the next year.

Extra-credit reading


DevOpsDays Austin — The provisioning panel starring JuJu, Crowbar, Puppet, Chef and Pallet

April 11, 2012

Last week DevOpsDays was held here in Austin.  It sold out in about day after it was announced and had a big waiting list.  The two-day event, which was held at National Instruments (who did an awesome job as host), featured talks and panels in the mornings and “open space” discussions in the afternoons.

The panel on the first day,  moderated by John Willis, was entitled: Provisioning Panel – Meet Juju, Crowbar, Puppet, Chef, Pallet + discussion.  After the panel I caught up with each of the members for a follow-up chat.  Here they are:

Juju – Mark Mimms of Canonical

Crowbar – Rob Booth of Zenoss

Puppet – Dan Bode of Puppet Labs

Chef – Matt Ray of Opscode

Pallet – Antoni Batchelli of Pallet Ops

Stay tuned for more DevOpsDays goodness in the days to come!

Extra-credit reading

Pau for now…


How to create a Basic or Advanced Crowbar build for Hadoop

November 29, 2011

As I mentioned in my previous entry, the code for the Hadoop barclamps is now available at our github repo.

To help you through the process, Crowbar lead architect Rob Hirschfeld has put together the two videos below.  The first, Crowbar Build (on cloud server), shows you how to use a cloud server to create a Crowbar ISO using the standard build process.  The second,  Advanced Crowbar Build (local) shows how to build a Crowbar v1.2 ISO using advanced techniques on a local desktop using a virtual machine.

Crowbar Build (on cloud server)

Advanced Crowbar Build (local)

Pau for now…


Open source Crowbar code now available for Hadoop

November 29, 2011

Earlier this month we announced that Dell would be open sourcing the Crowbar “barclamps” for Hadoop.  Well today is the day and the code is now available at our github repo.

Whats a Crowbar barclamp?

If you haven’t heard of project Crowbar it’s a software framework developed at Dell that started out as an installation tool for OpenStack.  As the project grew beyond installation to include monitoring capabilities, network discovery, performance data gathering etc., the developers behind it, Rob Hirschfeld and Greg Althaus, decided to rewrite it to allow modules to plug into the basic Crowbar functionality.  These modules or “barclamps” allow the framework to be used by a variety of projects.  Besides the OpenStack and Hadoop barclamps written by Dell, VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.

To help you get your bearings

As I mentioned in the opening  paragraph, the code for the Hadoop barclamp is now available.  To help you get started, below are a couple of videos that Rob put together.  The first walks you through how to install Crowbar and the second one explains how to use Crowbar to deploy Hadoop.

Extra-credit reading

Pau for  now…


Developers: How to get involved with Crowbar for Hadoop

November 8, 2011

In the previous entry I mentioned that we have developed and will be opensourcing “barclamps” (modules that sit on top of Crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop.  All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters.

If you would like to get involved, check out this 1 min video from Rob Hirschfeld talking about how:

Look for the code on the Crowbar GitHub repo by the last week of November.

Extra-credit reading:

Pau for now…


Crowbar: Where its been and where its going

October 24, 2011

Rob Hirschfeld, aka “Commander Crowbar,” recently posted a blog entry looking back at how Crowbar came to be, how its grown and where he hopes it will go from here.

What’s a Crowbar?

If you’re not familiar with Crowbar, its an open source software framework that began life as an installation tool to speed installation of OpenStack on Dell hardware.  The project incorporates the Opscode Chef Server tool and was originally created here at Dell by Rob and Greg Althaus.  Just four short months ago at OSCON 2011 the project took a big step forward when, along with the announcement of our OpenStack solution, we announced that we were opensourcing it.

DevOps-ilicous

As Rob points out in his blog, as we were delivering Crowbar as an installer a collective light bulb went off and we realized the role that Chef and tools like it play in a larger movement taking place in many Web shops today: the movement of DevOps.

The DevOps approach to deployment builds up systems in a layered model rather than using packaged images…Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.

On beyond installation and OpenStack

As the team began working more with Crowbar, it occurred to them that its use could be expanded in two ways: it could be used to do more than installation and it could be expanded to work with projects beyond OpenStack.

As for functionality, Crowbar now not only installs and configures but once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

The first project beyond OpenStack that we used Crowbar on was Hadoop.  In order to expand Crowbar’s usage we created the concept of  “barclamps” which are in essence modules that sit on top of the basic Crowbar functionality.  After we created the Hadoop barclamp, others picked up the charge and VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.

It takes a community

Crowbar development has recently been moved out into the open.  As Rob explains,

This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders.  Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github.

So what are you waiting for?  Join our mailing list, download the code or ISO, create a barclamp, make your voice heard.  Who’s next?

Extra-credit reading:

Pau for now…


Cloud Foundry picks up Crowbar to speed installation

August 17, 2011

In case you’re not familiar with Cloud Foundry, it’s an open source Platform as a Service project initiated at VMware.  More specifically it provides a platform for building, deploying, and running cloud apps using Spring for Java developers, Rails and Sinatra for Ruby developers, Node.js and other JVM frameworks including Grails.

The project began two years ago when VMware’s CEO Paul Maritz recruited Derek Collison and Mark Lucovsky out of Google and set them to working on Cloud Foundry.  Collison and Lucovsky, who built and maintained Google’s API services, were brought into leverage their experience of working with hugely scaled out architectures.

The Cloud Foundry project has only been public for a matter of months and one question that I’m sure has popped into your mind is what if I want to pilot Cloud Foundry in my own environment, won’t installation and configuration be a total pain?

Enter the Crowbar

Crowbar is an open source software framework developed at Dell to speed up the installation and configuration of open source cloud software onto bare metal systems.  By automating the process, Crowbar can reduce the time needed for installation from days to hours.

The software is modular in design so while the basic functionality is in Crowbar itself, “barclamps” sit on top of it to allow it work with a variety of projects.  The first use for crowbar was for OpenStack and the barclamp for that has been donated to the community.  Next came The Dell | Cloudera solution for Apache Hadoop and, just recently, Dreamhost announced that they currently working on a Ceph barclamp.  And now…

Two great tastes that taste great together

Today’s big news is that VMware is working with Dell to release and maintain a Crowbar barclamp that, in conjunction with Crowbar, will install and configure Cloud Foundry.  This capability, which will include multi-node configs over time, will allow organizations and service providers the ability to quickly and easily get pilots of Cloud Foundry up and running.

Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

If you’d like to try out Crowbar for yourself, check out: https://github.com/DellCloudEdge

Press added after initial posting

Extra-credit reading

Pau for now…


Dell announces availability of OpenStack solution; Open sources “Crowbar” software framework

July 26, 2011

Today at OSCON we are announcing the availability of the Dell OpenStack Cloud Solution along with the open sourcing of the code behind our Crowbar software framework.

The Solution

Dell has been a part of the OpenStack community since day one a little over a year ago and today’s news represents the first available cloud solution based on the OpenStack platform.  This Infrastructure-as-a-service solution includes a reference architecture based on Dell PowerEdge C servers, OpenStack open source software, the Dell-developed Crowbar software and services from Dell and Rackspace Cloud Builders.

Crowbar, keeping things short and sweet

Bringing up a cloud can be no mean feat, as a result a couple of our guys began working on a software framework that could be used to quickly (typically before coffee break!) bring up a multi-node OpenStack cloud on bare metal.   That framework became Crowbar.  What Crowbar does is manage the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in a matter of hours (or even minutes) instead of days.

Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

Code to the Community

As mentioned above, today Dell has released Crowbar to the community as open source code (you can get access to it the project’s GitHub site).  The idea is allow  users to build functionality to address their specific system needs.  Additionally we are working with the community to submit Crowbar as a core project in the OpenStack initiative.

Included in the Crowbar code contribution is the barclamp list, UI and remote API’s, automated testing scripts, build scripts, switch discovery, open source Chef server.  We are currently working with our legal team to determine how to release the BIOS and RAID which leverage third party components.  In the meantime since it is free (as in beer) software, although Dell cannot distribute it, users can directly go the vendors and download the components for free to get that functionality.

More Crowbar detail

For those who want some more detail, here are some bullets I’ve grabbed from Rob “Mr. Crowbar” Hirschfeld’s blog:

Important notes:

  • Crowbar uses Chef as it’s database and relies on cookbooks for node deployments
  • Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as “barclamps.”
  • Each barclamp has it’s own Chef configuration, UI subcomponent, deployment configuration, and documentation.

On the roadmap:

  • Hadoop support
  • Additional operating system support
  • Barclamp version repository
  • Network configuration
  • We’d like suggestions!  Please comment on Rob’s blog!

Extra-credit reading

Pau for now…


Structure: Learning about DevOps & Crowbar from Jesse Robbins

June 27, 2011

Last week on Day two of Structure the morning sessions ended with an  interesting discussion moderated by James Urquhart.  The session was entitled “DevOps – Reinventing the Developers Role in the Cloud Age” and featured Luke Kanies – CEO, Puppet Labs and Jesse Robbins – Co-Founder and CEO, Opscode.

After lunch I ran into Jesse and got him to sit down with me and provide some more insight into DevOps as well as explain what Opscode was doing with project Crowbar.

Some of the ground Jesse covers

  • (0:21) What is DevOps
  • (1:00) The shift that happens between developers and operations.  Writing code and getting it into production faster and how it shifts responsibilities between the two groups.
  • (2:52) Who are the prime targets for DevOps and how has this changed over time.
  • How DevOps began in web shops who needed to do things differently than legacy-bound enterprises.
  • How enterprises faced with greenfield opportunities are now embracing devops
  • (5:36) The crowbar installer which employs Opscode’s Chef and allows the rapid provisioning of an OpenStack cloud.

Extra-credit reading:


DevOps Days NYC: Mike Ainsworth of SaltStack

January 7, 2014

Infrastructure automation and configuration management have become a hot topic as organizations, particularly those in the web and cloud space have continued to scale out. The first player, that helped define the space, was CFengine.  Heavy weights in this space now include Puppet and Chef, and at Dell we lead, Project Crowbar, an open source effort in this area.

The new kid on the block, based in Salt Lake city, is SaltStack.   Although relatively new, they count LinkedIn as one of their users.   At DevOps Days NYC back in October I grabbed some time with Salt solutions engineer, Mike Ainsworth.  Here is what Mike had to say:

Some of the ground Mike covers

  • What does SaltStack do
  • When did they get their start and how has their momentum been
  • How they differ from the other players in the space
  • How Linkedin is using SaltStack
  • The technical details re what they are focusing on going forward

Extra-credit reading

Pau for now…


Talking to the Docker Dudes

September 12, 2013

This morning a group of us here at Dell met with Ben Golub, Jerome Petazzoni and Nick Stinemates of dotCloud, the company behind the wildly popular open source project, Docker, “the Linux container engine.”  They came to sample the great barbecue and to chat about how Docker might potentially work with Project Sputnik, the Crowbar Project and a few other efforts.

Docker, which went live in March already has 150 contributors, 60,000+ downloads and 1000s of applications containerized and uploaded to their registry.   Given the fact that the company only has 18 employees, quite a bit of this work has been done by the passionate community that has formed in the first six months.

Overview and Tech talk

I did two interviews with the gents from Docker, a higher level overview with Ben their CEO and a more technical talk with SRE manager Jerome and Nick, their sales and deployment engineer.  Enjoy!

Some of the ground Ben covers:

  • What is Docker?
  • How it developed out of dotCloud’s PaaS efforts
  • How Ben got involved with the project and his background
  • What are dotCloud’s plans for Docker and who is integrating with it?

Some of the ground Jerome and Nick cover:

  • How long they’ve been involved and what they focus on
  • How Docker works with LXC and how it might work without LXC
  • Ubuntu is recommended but all you need is AUFS support
  • In next release they plan to offer official support beyond Ubuntu
  • Holy DevOps batman, Docker has something to offer Devs, QA Engineers, Continuos  integration and Sys Ops.

Extra-credit reading

Pau for now…


Project Sputnik now comes with 3 month free trial on Joyent cloud

July 23, 2013

Joyent logoAs of today we are making available three months of free use of the Joyent Cloud to owners of the XPS 13 developer edition.

The idea behind Project Sputnik, has always been to provide a client-to-cloud platform for developers and today we are offering access to the Joyent Cloud to complete the solution.

What you get and how you get it

With the trial you get either two g3-standard-0.625-kvm instances running Ubuntu for 3 months or one g3-standard-1.75-kvm instance running Ubuntu for 3 months.

We will be setting up a landing page in the next day or two provide elegant access to the Joyent Cloud but for those who want to get started right away you can simply follow the “How do I get Started” instructions below.  We are kicking this off to begin with with 500 free accounts, first come first served.

3 components wJoyentProfile Tool and Cloud Launcher

Also available now are the Project Sputnik Cloud Launcher and profile tool.   The profile tool is designed to provide access to a library of community-created profiles, and to configure and quickly set up development environments and tool chains.  Today we have three sample profiles available: Emacs, Ruby and JavaScript.  Documentation on how to create a profile will be coming soon so stay tuned.

The cloud launcher creates a seamless link from the client to the cloud, to facilitate ongoing development of application environments.  There is a Juju version of the launcher that currently comes with Sputnik and today we are announcing a version that Opscode has developed which uses spiceweasel as its underlying library.  You can check out a demo of it here.  We are also working to connect the chef version of the cloud launcher to the Joyent trial, more to come on that soon.

But wait, there’s more

In related Dell Open Source news we’ve got a whole lot of momentum going on.  You can check out all the news in today’s press release but here are the highlights:

Dell OpenStack-Powered Cloud Solution

Now available with: OpenStack Grizzly support, support for Dell Multi-Cloud Manager (formerly Enstratius), and extended reference architecture support, including the Dell PowerEdge C8000

Dell Cloud Transformation Services

The new consulting services provide assistance with assessing, building, operating and running cloud environments, and enable and accelerate enterprise OpenStack adoption.

Dell Cloudera Hadoop Solution

Now supports the newest version of Cloudera Enterprise. Updates allow customers to perform real-time SQL interactive queries and Hadoop-based batch processing, simplifying the process of querying data in Hadoop environments.

Intel Distribution for Apache Hadoop

Dell has tested and certified the Intel Distribution for Apache Hadoop on Dell PowerEdge servers. Additionally Dell Solution Centers validated the reference architecture and developed a technical whitepaper that simplifies the deployment of Intel Distribution on the Dell platform

 Crowbar

Dell has released RAID and BIOS configuration capabilities to the Crowbar open source community.  SUSE has integrated Crowbar functionality as part of SUSE Cloud to make OpenStack-based private cloud deployments seamless.

Dasein open source project

Dell confirmed its commitment to further develop and support the Dasein open source project, as pioneered by recently acquired Enstratius.

Phew, a whole lot of shaking going on! :-)

===========================================

How do I get Started with Joyent Cloud trial

Step 1:

Open a terminal window press Ctrl + Alt +T

1.1. $ wget https://us-east.manta.joyent.com/jens/public/sputnik.tar

1.2. $ sudo tar -C / -xvf sputnik.tar

Step 2:

Find and run the “Install Joyent Public Could” in the launcher.

Look for the big Joyent LOGO.

Step 3:

Signup for a free trial account on the Joyent Public Cloud.

Open Firefox, goto http://www.joyent.com

Step 4:

Back in the terminal window, type the following command:

$ /usr/share/applications/joyentInstaller.sh

Step 5:

5.1. $ wget -O key-generator.sh https://us-east.manta.joyent.com/jens/public/key-generator.sh

5.2. $ chmod 755 key-generator.sh

5.3. $ ./key-generator.sh (enter you username and password for the jpc)

To source your new environment variables run the following commands

5.4. source ~/.bash_profile

Step 6:

6.1   To Confirm that the Joyent cloud SDK is installed:   $ sdc-listdatacenters

6.2   To confirm that the Joyent Manta SDK is installed:   $ mls /manta/public/sdks

How do I provision a new instance?

Sign in to the Joyent portal and click the  in the upper right portion of the screen. Once you’re there, the tool will walk you through the choice of datacenters, images, and instance types and sizes. You’ll have a chance to review the hourly and monthly cost of the instance, and provide a memorable name for the instance. Once you’ve decided on the type of instance that fits your project, click the  button and the system will ask to confirm your request. The provisioning will start immediately, but may take a few seconds to complete. Clicking on the new named instance will show its assigned public IP address when provisioning is complete. You may SSH into the instance with ssh -l root <ip address>.

How do I stop, resize or reboot instances?

Shutting down, resizing or rebooting your instance can all be executed through the customer portal of Joyent. In addition, we’ve provided a script you can use to perform these steps within your instance.

How do I install software on my instance?

To install or update software on your instance, you’ll need to run commands as either the administrative or root user of your instance. For tips on how to run commands and installation processes, check out the pages on how to install software on your instance.

How do I secure my instance?

Joyent take cloud security very seriously and we have refined many processes to reduce risk and preserve the integrity of data managed in your instance. For a full list of security checks and processes, please visit the security center in our documentation.

How do you manage your instance resource usage?

One of strengths of Joyent is the ability to have full and detailed transparency of every aspect of your infrastructure and application. You can use Cloud Analytics to provide you real-time, diagnostic heatmaps of system behavior. In addition, using these tips here can provide you better control over optimizing the performance of your instance.

How do you manage a database on your instance?

Instances on Joyent can be pre-configured to run a wide range of databases and database services. Joyent supports: MySQL, Percona, Riak, MongoDB, as well as integration to database services from companies like, Cloudant or MongoLab. For big data projects, Joyent is an ideal platform for configuring and running a Hadoop cluster. Check out these guides on how to set up a database or configure your Hadoop cluster.

How do you analyze performance of my instances?

Joyent is the best cloud in the industry for monitoring the entire health of your stack. Using Cloud Analytics, you have the ability to examine, in real-time, the performance characteristics of every level of your application, and network. If you just want to perform server level monitoring, we’ve built integration with leading monitoring tools from New Relic and Nodefly as well.

Where can I learn more?

Our documentation center and engineering blogs are terrific resources for you to learn more about Joyent and participate in the Joyent community. The Dev Center resources we’ve built for you will hopefully get you started on a path to success with Joyent. For additional help or training, please visit:

Pau for now…


Dell and Sputnik go to OSCON

July 18, 2013

Next week, myself, Michael Cote and a whole other bunch of Dell folk will be heading out to Portland for the 15th annual OSCON-ana-polooza.  We will have two talks that you might want to check out:

Cote and I will be giving the first and the second will be lead by Joseph George and James Urquhart.

Sputnik Shirt

And speaking of Project Sputnik, we will be giving away three of our XPS 13 developer editions:  one as a door prize at the OpenStack birthday party, one as a drawing at our booth and one to be given away at James and Joseph’s talk listed above.

We will also have a limited amount of the shirt to the right so stop by the booth.

But wait, there’s more….

To learn firsthand about Dell’s open source solutions be sure to swing by booth #719 where we will have experts on hand to talk to you about our wide array of solutions:

  • OpenStack cloud solutions
  • Hadoop big data solutions
  • Crowbar
  • Project Sputnik (the client to cloud developer platform)
  • Dell Multi-Cloud Manager (the platform formerly known as “Enstratius”)
  • Hyperscale computing systems

Hope to see you there.

Pau for now…


Copper: Dell goes out ARMed

May 29, 2012

We’ve been watching the ARM market develop over the past few years as these highly efficient chips that have been driving tablets and cell phones have been finding their way more and more into hyperscale servers.   Well watch no more, today were are sallying forth.  Why now?  Because some of our biggest customers have told us that they felt the time is now to start working with these low powered, highly efficient chips for their servers.

HW + SW = Solution

Today we announced that we will be shipping the new Dell “Copper” ARM servers via a seed unit program to select hyperscale customers worldwide.  But a server does not an ecosystem make so we are doing what we can to help partners and developers get started building out applications for the platform.  Given that two of the key areas where the extreme efficiencies of ARM play particularly well are Web front-ends and Hadoop environments, we have “ARMed” key partners like Canonical and Cloudera with units.

Early days

At this point it is still early days in the world of ARM servers so we designed Copper specifically for developers and customers to create code and test performance, not for production.  To help developers get started we have struck a partnership between the Dell Solutions Centers and Texas Advanced Computing Center (TACC) to provide devs with remotely accessible clusters to develop and collaborate on.  And speaking of developers,  Dells own devs are working to deliver an ARM-based version of our open source infrastructure management software, Crowbar.

Speeds and Feeds

And in case your wondering about the specs of the hardware:

  • Dell Copper servers are a shared infrastructure design, which allows easy deployment and reconfiguration of the sleds.
  • Each ARM server node draws about 15 watts max power, so the total power draw for a full chassis is less than 750 watts.
  • The server nodes discover themselves and interconnect when deployed, so workloads can easily run across the entire 48 nodes.
  • And it’s still powerful, with four ARM server nodes per sled, and 12 total sleds, bringing a total of 48 server nodes to a single 3U C5000 chassis.

Stay tuned for more…

Extra-credit reading

Pau for now…


On beyond North America — Dell’s OpenStack solution now available in Europe and Asia

March 21, 2012

Last summer at OSCON Dell announced the availability of our OpenStack solution in the US and Canada.  Today at World Hosting Days in Rust Germany we are now announcing that our OpenStack-Powered Cloud Solution is available in Europe and Asia.

If you’re not familiar with it, OpenStack is an open source cloud project built on a foundation of code initially donated by NASA and Rackspace.  The project kicked off a little over a year and a half ago here in Austin and it has gained amazing traction since then.

Dell’s offering

Dell’s OpenStack cloud offering is an open source, on premise cloud solution based on the OpenStack platform running on Ubuntu.  Its composed of:

  • The OpenStack cloud operating system
  • PowerEdgeC servers: C6100, C6105, C2100 and, coming soon, Dell’s new C6220 and R720
  • The Crowbar deployment and management software framework – developed and coded by Dell :)
  • Dell’s OpenStack reference architecture
  • Dell Services

Crowbar software framework

To give a little more  background on the Crowbar software framework, its an open source project developed initially at Dell and you can grab it off github.  The framework, which is under the Apache 2.0 license,  manages the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in hours, as opposed to days.

Once the initial deployment is complete, you can use Crowbar to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.   Beyond Dell, companies like VMware, Dreamhost and Zenoss have built “barclamps”  that allow them to utilize Crowbar’s modular design.  Additionally, customers who buy the Dell OpenStack-Powered Cloud Solution get training, deployment, and support on Crowbar.

So as of today, customers in the UK, Germany and China can purchase the Dell OpenStack-Powered Cloud Solution.  As customer demand grows in other regions we will be adding more countries so stay tuned.  If the first 18 mos of the project are any indication of whats the pace is like to come, we are all going to be in for a lot more excitement.

For more info, email: OpenStack@Dell.com

Extra-credit reading

Pau for now…


Dell’s Big Data escalator pitch

February 24, 2012

At our sales kickoff in Vegas, Rob Hirschfeld chose a unique vehicle to succinctly convey our Big Data story here at Dell.  Check out the video below to hear one of our chief software architects for our Big Data and OpenStack solutions explain, in less than 90 seconds, what we are up to in the space and the value it brings customers.

Extra credit reading

Pau for now…


Hadoop World, a belated summary

February 13, 2012

With O’Reilly’s big data conference Strata coming up in just a couple of weeks, I thought I might as well get around to finally writing up my notes from Hadoop World .  The event, which was put on by Cloudera, was held last November 8-9 in New York city.   There were over 1,400 attendees from 580 companies and 27 countries with two thirds of the audience being technical.

Growing beyond geek fest

The event itself has picked up significant momentum over the last three years going from 500 attendees, to 900 the second year, to over 1400 this past year.  The tone has gone from geek-fest to an event focused also on business problems e.g. one of the keynotes was by Larry Feinsmith, managing director of the office of the CIO at JP Morgan Chase.  Besides Dell, other large companies like HP, Oracle and Cisco also participated.

As a platinum sponsor, Dell  had both a booth and a technical presentation.   At the event we announced that we would be open sourcing the Crowbar barclamp for Hadoop and at out booth we showed off the Dell | Hadoop Big Data Solution which is based on Cloudera Enterprise.

Cutting’s observations

Doug Cutting, the father of  Hadoop, Cloudera employee and chairman of the Apache software foundation, gave a much anticipated keynote.  Here are some of the key things I caught:

  • Still young: While Cutting felt that Hadoop had made tremendous progress he saw it as still young with lots of missing parts and niches to be filled.
  • Big Top: He talked about the Apache “Bigtop” project which is an open source program to pull together the various pieces of the Hadoop ecosystem.  He explained that Bigtop is intended to serve as the basis for the Cloudera Distribution of Hadoop (CDH), much the same way Fedora is the basis  for RHEL (Redhat Enterprise Linux).
  • “Hadoop” as “Linux“: Cutting also talked about how Hadoop has become the kernel of the distributed OS for big data.  He explained that, much the same way that “Linux” is technically only the kernel of the GNU Linux operating system, people are using the word Hadoop to mean the entire Hadoop ecosystem including utilities.

Interviews from the event

To get more of the flavor of the event here is a series of interviews I conducted at the show, plus one where I got the camera turned on me:

Extra-credit reading

Blogs regarding Dell’s crowbar announcement

Hadoop Glossary

  • Hadoop ecosystem
    • Hadoop: An open source platform, developed at Yahoo that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.  It is particularly suited to large volumes of unstructured data such as Facebook comments and Twitter tweets, email and instant messages, and security and application logs.
    • MapReduce: a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in parallel on large clusters of commodity hardware in a reliable, fault-tolerant manner.  Hadoop acts as a platform for executing MapReduce.  MapReduce came out of Google
    • HDFS: Hadoop’s Distributed File system allows large application workloads to be broken into smaller data blocks that are replicated and distributed across a cluster of commodity hardware for faster processing.
  • Major Hadoop utilities:
    • HBase: The Hadoop database that supports structured data storage for large tables.   It provides real time read/write access to your big data.
    • Hive:  A data warehousing solution built on top of Hadoop.  An Apache project
    • Pig: A platform for analyzing large data that leverages parallel computation.  An Apache project
    • ZooKeeper:  Allows Hadoop administrators to track and coordinate distributed applications.  An Apache project
    • Oozie: a workflow engine for Hadoop
    • Flume: a service designed to collect data and put it into your  Hadoop environment
    • Whirr: a set of libraries for running cloud services.  It’s ideal for running temporary Hadoop clusters to carry out a proof of concept, or to run a few one-time jobs.
    • Sqoop: a tool designed to transfer data between Hadoop and relational databases.  An Apache project
    • Hue: a browser-based desktop interface for interacting with Hadoop

Web Glossary part three: Infrastructure tier

January 24, 2012

This is the last in my three-part Web Glossary series.  As I previously explained, in compiling this I pulled information from various and sundry sources across the Web including Wikipedia, community and company web sites and the brain of Cote.

The idea behind the glossary is to help our teams get a better understand of the wild and wacky world of the Web and Web developers as we move forward with our Web|Tech vertical.  I figured I might as also share it with a few friends.

Today’s focus, having worked our way down from the top, is the infrastructure tier (with a short catch-all bucket at the end , “Misc.”)

Infrastructure

General Terms

  • DevOps:  The goal of the DevOps movement is to drive out inefficiency in web shops by bridging the gap (and lessening conflict) between traditional development activity and operations activity.  It seeks to address this issue by providing tools and practices to bring these two groups closer together and provide for greater automation of processes.  Key tools in this effort are Opscode’s Chef and Puppet lab’s Puppet which automate the set-up and management of infrastructure.
  • PUE: Power Usage Effectiveness is a measure of how efficiently a computer data center uses its power; specifically, how much of the power is actually used by the computing equipment (in contrast to cooling and other overhead).   PUE is the ratio of total amount of power used by a computer data center facility to the power delivered to computing equipment.  The closer to 1.0, the better the PUE.
  • Distributed management: refers to the setup, provisioning, maintenance and management of the scale-out infrastructure (either physical or virtual) that has historically been characteristic of web firms and is increasing typical within traditional enterprise customers.  This includes players like Chef and Puppet for provisioning and configuration, New Relic and Splunk for monitoring and management, and Loggly/Eucalyptus/OpenStack/ VMware for management monitoring.

Projects/Entities

  • CrowbarCrowbar is a Dell-developed open source software framework designed to speed up the installation and configuration of open source cloud software onto bare metal systems.  By automating the process, Crowbar can reduce the time needed for installation from days to hours.  The software is modular in design so while the basic functionality is in Crowbar itself, “barclamps” sit on top of it to allow it work with a variety of projects.  There have been barclamps built for OpenStack, Hadoop, CloudFoundry and Dreamhost.
  • Ubuntu:  The most popular desktop linux distribution.  On the server side they are supporting OpenStack and have an offering called the Ubuntu Enterprise Cloud.   Backed by the commercial company Canonical.
  • Puppet: a configuration management tool designed to automate the set up and management of infrastructure.  A key DevOps tool.  It is produced by Puppet labs
  • Chef: a configuration management tool designed to automate the set up and management of infrastructure.  A key DevOps tool.  It is produced by Opscode, who hosts a cloud-based version of Chef called the Opscode Platform.
  • Nagios: a popular open source computer system and network monitoring software application. It watches hosts and services, alerting users when things go wrong and again when they get better.
  • Ganglia: an open source scalable distributed monitoring system for high-performance computing systems such as clusters and grids.

Misc

  • LAMP stack:  Open source stack that provides a viable general purpose web server.  The name comes from the first letters of its components: Linux, Apache web server, MySQL and PHP (or Perl or Python).   LAMP has become a de facto development standard and is an excellent example of how open source software has made its way into enterprise environments through unofficial channels.
  • Apache Software Foundation: A decentralized group of developers that produce open source software under the Apache license.  Notable projects include: Apache web server, Hadoop, CouchDB, Cassandra, Tomcat, Subversion
  • Nginx: an open source web server that recently has been gaining considerable traction
  • Recipes:  They encapsulate collections of software resources which are executed in the order defined to configure a system.

Extra-credit reading

Pau for now…


Opscode visits

January 10, 2012

This afternoon Matt Ray, Technical Evangelist for Opscode, stopped by Dell’s Round Rock HQ to brief a gaggle of folks on what they are up to.  Cote arranged the visit as well as one last month with Puppet labs, which I unfortunately wasn’t able to make.

After Matt, with some help from teammates on the phone, briefed the Dell gang I grabbed some time with him to get the 5 minute Reader’s Digest version.  Here is the result.

Some of the ground Matt covers:

  • What are Opscode and Chef?
  • How did they come to be?
  • The hosted version of Chef (moving from EC2 to Rackspace)
  • Crowbar: lending a helping hand
  • What’s next for Opscode and what do they have up their sleeve for 2012?

Extra-credit reading

Pau for now…


Hadoop World: What Dell is up to with Big Data, Open Source and Developers

December 18, 2011

Besides interviewing a bunch of people at Hadoop World, I also got a chance to sit on the other side of the camera.  On the first day of the conference I got a slot on SiliconANGLE’s the Cube and was interviewed by Dave Vellante, co-founder of Wikibon and John Furrier, founder of SiliconANGLE.

-> Check out the video here.

Some of the ground we cover

  • How Dell got into the cloud/scale-out arena and how that lead us to Big Data
  • (2:08) The details behind the Dell|Cloudera solution for Apache Hadoop and our “secret sauce,” project crowbar.
  • (4:00) Dell’s involvement in and affinity for open source software
  • (5:31) Dell’s interest in and strategy around courting developers
  • (7:35) Dell’s strategy of Make, Partner or Buy in the cloud space
  • (11:10) How real is OpenStack and how is it evolving.

Extra-credit reading

Pau for now…


Follow

Get every new post delivered to your Inbox.

Join 88 other followers

%d bloggers like this: