Project Sputnik now comes with 3 month free trial on Joyent cloud

July 23, 2013

Joyent logoAs of today we are making available three months of free use of the Joyent Cloud to owners of the XPS 13 developer edition.

The idea behind Project Sputnik, has always been to provide a client-to-cloud platform for developers and today we are offering access to the Joyent Cloud to complete the solution.

What you get and how you get it

With the trial you get either two g3-standard-0.625-kvm instances running Ubuntu for 3 months or one g3-standard-1.75-kvm instance running Ubuntu for 3 months.

We will be setting up a landing page in the next day or two provide elegant access to the Joyent Cloud but for those who want to get started right away you can simply follow the “How do I get Started” instructions below.  We are kicking this off to begin with with 500 free accounts, first come first served.

3 components wJoyentProfile Tool and Cloud Launcher

Also available now are the Project Sputnik Cloud Launcher and profile tool.   The profile tool is designed to provide access to a library of community-created profiles, and to configure and quickly set up development environments and tool chains.  Today we have three sample profiles available: Emacs, Ruby and JavaScript.  Documentation on how to create a profile will be coming soon so stay tuned.

The cloud launcher creates a seamless link from the client to the cloud, to facilitate ongoing development of application environments.  There is a Juju version of the launcher that currently comes with Sputnik and today we are announcing a version that Opscode has developed which uses spiceweasel as its underlying library.  You can check out a demo of it here.  We are also working to connect the chef version of the cloud launcher to the Joyent trial, more to come on that soon.

But wait, there’s more

In related Dell Open Source news we’ve got a whole lot of momentum going on.  You can check out all the news in today’s press release but here are the highlights:

Dell OpenStack-Powered Cloud Solution

Now available with: OpenStack Grizzly support, support for Dell Multi-Cloud Manager (formerly Enstratius), and extended reference architecture support, including the Dell PowerEdge C8000

Dell Cloud Transformation Services

The new consulting services provide assistance with assessing, building, operating and running cloud environments, and enable and accelerate enterprise OpenStack adoption.

Dell Cloudera Hadoop Solution

Now supports the newest version of Cloudera Enterprise. Updates allow customers to perform real-time SQL interactive queries and Hadoop-based batch processing, simplifying the process of querying data in Hadoop environments.

Intel Distribution for Apache Hadoop

Dell has tested and certified the Intel Distribution for Apache Hadoop on Dell PowerEdge servers. Additionally Dell Solution Centers validated the reference architecture and developed a technical whitepaper that simplifies the deployment of Intel Distribution on the Dell platform

 Crowbar

Dell has released RAID and BIOS configuration capabilities to the Crowbar open source community.  SUSE has integrated Crowbar functionality as part of SUSE Cloud to make OpenStack-based private cloud deployments seamless.

Dasein open source project

Dell confirmed its commitment to further develop and support the Dasein open source project, as pioneered by recently acquired Enstratius.

Phew, a whole lot of shaking going on! :-)

===========================================

How do I get Started with Joyent Cloud trial

Step 1:

Open a terminal window press Ctrl + Alt +T

1.1. $ wget https://us-east.manta.joyent.com/jens/public/sputnik.tar

1.2. $ sudo tar -C / -xvf sputnik.tar

Step 2:

Find and run the “Install Joyent Public Could” in the launcher.

Look for the big Joyent LOGO.

Step 3:

Signup for a free trial account on the Joyent Public Cloud.

Open Firefox, goto http://www.joyent.com

Step 4:

Back in the terminal window, type the following command:

$ /usr/share/applications/joyentInstaller.sh

Step 5:

5.1. $ wget -O key-generator.sh https://us-east.manta.joyent.com/jens/public/key-generator.sh

5.2. $ chmod 755 key-generator.sh

5.3. $ ./key-generator.sh (enter you username and password for the jpc)

To source your new environment variables run the following commands

5.4. source ~/.bash_profile

Step 6:

6.1   To Confirm that the Joyent cloud SDK is installed:   $ sdc-listdatacenters

6.2   To confirm that the Joyent Manta SDK is installed:   $ mls /manta/public/sdks

How do I provision a new instance?

Sign in to the Joyent portal and click the  in the upper right portion of the screen. Once you’re there, the tool will walk you through the choice of datacenters, images, and instance types and sizes. You’ll have a chance to review the hourly and monthly cost of the instance, and provide a memorable name for the instance. Once you’ve decided on the type of instance that fits your project, click the  button and the system will ask to confirm your request. The provisioning will start immediately, but may take a few seconds to complete. Clicking on the new named instance will show its assigned public IP address when provisioning is complete. You may SSH into the instance with ssh -l root <ip address>.

How do I stop, resize or reboot instances?

Shutting down, resizing or rebooting your instance can all be executed through the customer portal of Joyent. In addition, we’ve provided a script you can use to perform these steps within your instance.

How do I install software on my instance?

To install or update software on your instance, you’ll need to run commands as either the administrative or root user of your instance. For tips on how to run commands and installation processes, check out the pages on how to install software on your instance.

How do I secure my instance?

Joyent take cloud security very seriously and we have refined many processes to reduce risk and preserve the integrity of data managed in your instance. For a full list of security checks and processes, please visit the security center in our documentation.

How do you manage your instance resource usage?

One of strengths of Joyent is the ability to have full and detailed transparency of every aspect of your infrastructure and application. You can use Cloud Analytics to provide you real-time, diagnostic heatmaps of system behavior. In addition, using these tips here can provide you better control over optimizing the performance of your instance.

How do you manage a database on your instance?

Instances on Joyent can be pre-configured to run a wide range of databases and database services. Joyent supports: MySQL, Percona, Riak, MongoDB, as well as integration to database services from companies like, Cloudant or MongoLab. For big data projects, Joyent is an ideal platform for configuring and running a Hadoop cluster. Check out these guides on how to set up a database or configure your Hadoop cluster.

How do you analyze performance of my instances?

Joyent is the best cloud in the industry for monitoring the entire health of your stack. Using Cloud Analytics, you have the ability to examine, in real-time, the performance characteristics of every level of your application, and network. If you just want to perform server level monitoring, we’ve built integration with leading monitoring tools from New Relic and Nodefly as well.

Where can I learn more?

Our documentation center and engineering blogs are terrific resources for you to learn more about Joyent and participate in the Joyent community. The Dev Center resources we’ve built for you will hopefully get you started on a path to success with Joyent. For additional help or training, please visit:

Pau for now…


Dell and Sputnik go to OSCON

July 18, 2013

Next week, myself, Michael Cote and a whole other bunch of Dell folk will be heading out to Portland for the 15th annual OSCON-ana-polooza.  We will have two talks that you might want to check out:

Cote and I will be giving the first and the second will be lead by Joseph George and James Urquhart.

Sputnik Shirt

And speaking of Project Sputnik, we will be giving away three of our XPS 13 developer editions:  one as a door prize at the OpenStack birthday party, one as a drawing at our booth and one to be given away at James and Joseph’s talk listed above.

We will also have a limited amount of the shirt to the right so stop by the booth.

But wait, there’s more….

To learn firsthand about Dell’s open source solutions be sure to swing by booth #719 where we will have experts on hand to talk to you about our wide array of solutions:

  • OpenStack cloud solutions
  • Hadoop big data solutions
  • Crowbar
  • Project Sputnik (the client to cloud developer platform)
  • Dell Multi-Cloud Manager (the platform formerly known as “Enstratius”)
  • Hyperscale computing systems

Hope to see you there.

Pau for now…


DevOpsDays: Crowbar, where its been and where its going

April 25, 2012

Earlier this month at DevOpsDays here in Austin the Dell Crowbar crew hosted a session and gave a demo.  If you’re not familiar with it, Crowbar is an open source software framework written at Dell.  I grabbed some time with Crowbar architect Rob Hirschfeld and got him to recap how far we’ve come in its less than a year and where he sees us going over the next year.

Extra-credit reading


DevOpsDays Austin — The provisioning panel starring JuJu, Crowbar, Puppet, Chef and Pallet

April 11, 2012

Last week DevOpsDays was held here in Austin.  It sold out in about day after it was announced and had a big waiting list.  The two-day event, which was held at National Instruments (who did an awesome job as host), featured talks and panels in the mornings and “open space” discussions in the afternoons.

The panel on the first day,  moderated by John Willis, was entitled: Provisioning Panel – Meet Juju, Crowbar, Puppet, Chef, Pallet + discussion.  After the panel I caught up with each of the members for a follow-up chat.  Here they are:

Juju – Mark Mimms of Canonical

Crowbar – Rob Booth of Zenoss

Puppet – Dan Bode of Puppet Labs

Chef – Matt Ray of Opscode

Pallet – Antoni Batchelli of Pallet Ops

Stay tuned for more DevOpsDays goodness in the days to come!

Extra-credit reading

Pau for now…


On beyond North America — Dell’s OpenStack solution now available in Europe and Asia

March 21, 2012

Last summer at OSCON Dell announced the availability of our OpenStack solution in the US and Canada.  Today at World Hosting Days in Rust Germany we are now announcing that our OpenStack-Powered Cloud Solution is available in Europe and Asia.

If you’re not familiar with it, OpenStack is an open source cloud project built on a foundation of code initially donated by NASA and Rackspace.  The project kicked off a little over a year and a half ago here in Austin and it has gained amazing traction since then.

Dell’s offering

Dell’s OpenStack cloud offering is an open source, on premise cloud solution based on the OpenStack platform running on Ubuntu.  Its composed of:

  • The OpenStack cloud operating system
  • PowerEdgeC servers: C6100, C6105, C2100 and, coming soon, Dell’s new C6220 and R720
  • The Crowbar deployment and management software framework – developed and coded by Dell :)
  • Dell’s OpenStack reference architecture
  • Dell Services

Crowbar software framework

To give a little more  background on the Crowbar software framework, its an open source project developed initially at Dell and you can grab it off github.  The framework, which is under the Apache 2.0 license,  manages the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in hours, as opposed to days.

Once the initial deployment is complete, you can use Crowbar to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.   Beyond Dell, companies like VMware, Dreamhost and Zenoss have built “barclamps”  that allow them to utilize Crowbar’s modular design.  Additionally, customers who buy the Dell OpenStack-Powered Cloud Solution get training, deployment, and support on Crowbar.

So as of today, customers in the UK, Germany and China can purchase the Dell OpenStack-Powered Cloud Solution.  As customer demand grows in other regions we will be adding more countries so stay tuned.  If the first 18 mos of the project are any indication of whats the pace is like to come, we are all going to be in for a lot more excitement.

For more info, email: OpenStack@Dell.com

Extra-credit reading

Pau for now…


Hadoop World, a belated summary

February 13, 2012

With O’Reilly’s big data conference Strata coming up in just a couple of weeks, I thought I might as well get around to finally writing up my notes from Hadoop World .  The event, which was put on by Cloudera, was held last November 8-9 in New York city.   There were over 1,400 attendees from 580 companies and 27 countries with two thirds of the audience being technical.

Growing beyond geek fest

The event itself has picked up significant momentum over the last three years going from 500 attendees, to 900 the second year, to over 1400 this past year.  The tone has gone from geek-fest to an event focused also on business problems e.g. one of the keynotes was by Larry Feinsmith, managing director of the office of the CIO at JP Morgan Chase.  Besides Dell, other large companies like HP, Oracle and Cisco also participated.

As a platinum sponsor, Dell  had both a booth and a technical presentation.   At the event we announced that we would be open sourcing the Crowbar barclamp for Hadoop and at out booth we showed off the Dell | Hadoop Big Data Solution which is based on Cloudera Enterprise.

Cutting’s observations

Doug Cutting, the father of  Hadoop, Cloudera employee and chairman of the Apache software foundation, gave a much anticipated keynote.  Here are some of the key things I caught:

  • Still young: While Cutting felt that Hadoop had made tremendous progress he saw it as still young with lots of missing parts and niches to be filled.
  • Big Top: He talked about the Apache “Bigtop” project which is an open source program to pull together the various pieces of the Hadoop ecosystem.  He explained that Bigtop is intended to serve as the basis for the Cloudera Distribution of Hadoop (CDH), much the same way Fedora is the basis  for RHEL (Redhat Enterprise Linux).
  • “Hadoop” as “Linux“: Cutting also talked about how Hadoop has become the kernel of the distributed OS for big data.  He explained that, much the same way that “Linux” is technically only the kernel of the GNU Linux operating system, people are using the word Hadoop to mean the entire Hadoop ecosystem including utilities.

Interviews from the event

To get more of the flavor of the event here is a series of interviews I conducted at the show, plus one where I got the camera turned on me:

Extra-credit reading

Blogs regarding Dell’s crowbar announcement

Hadoop Glossary

  • Hadoop ecosystem
    • Hadoop: An open source platform, developed at Yahoo that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.  It is particularly suited to large volumes of unstructured data such as Facebook comments and Twitter tweets, email and instant messages, and security and application logs.
    • MapReduce: a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in parallel on large clusters of commodity hardware in a reliable, fault-tolerant manner.  Hadoop acts as a platform for executing MapReduce.  MapReduce came out of Google
    • HDFS: Hadoop’s Distributed File system allows large application workloads to be broken into smaller data blocks that are replicated and distributed across a cluster of commodity hardware for faster processing.
  • Major Hadoop utilities:
    • HBase: The Hadoop database that supports structured data storage for large tables.   It provides real time read/write access to your big data.
    • Hive:  A data warehousing solution built on top of Hadoop.  An Apache project
    • Pig: A platform for analyzing large data that leverages parallel computation.  An Apache project
    • ZooKeeper:  Allows Hadoop administrators to track and coordinate distributed applications.  An Apache project
    • Oozie: a workflow engine for Hadoop
    • Flume: a service designed to collect data and put it into your  Hadoop environment
    • Whirr: a set of libraries for running cloud services.  It’s ideal for running temporary Hadoop clusters to carry out a proof of concept, or to run a few one-time jobs.
    • Sqoop: a tool designed to transfer data between Hadoop and relational databases.  An Apache project
    • Hue: a browser-based desktop interface for interacting with Hadoop

Opscode visits

January 10, 2012

This afternoon Matt Ray, Technical Evangelist for Opscode, stopped by Dell’s Round Rock HQ to brief a gaggle of folks on what they are up to.  Cote arranged the visit as well as one last month with Puppet labs, which I unfortunately wasn’t able to make.

After Matt, with some help from teammates on the phone, briefed the Dell gang I grabbed some time with him to get the 5 minute Reader’s Digest version.  Here is the result.

Some of the ground Matt covers:

  • What are Opscode and Chef?
  • How did they come to be?
  • The hosted version of Chef (moving from EC2 to Rackspace)
  • Crowbar: lending a helping hand
  • What’s next for Opscode and what do they have up their sleeve for 2012?

Extra-credit reading

Pau for now…


Hadoop World: What Dell is up to with Big Data, Open Source and Developers

December 18, 2011

Besides interviewing a bunch of people at Hadoop World, I also got a chance to sit on the other side of the camera.  On the first day of the conference I got a slot on SiliconANGLE’s the Cube and was interviewed by Dave Vellante, co-founder of Wikibon and John Furrier, founder of SiliconANGLE.

-> Check out the video here.

Some of the ground we cover

  • How Dell got into the cloud/scale-out arena and how that lead us to Big Data
  • (2:08) The details behind the Dell|Cloudera solution for Apache Hadoop and our “secret sauce,” project crowbar.
  • (4:00) Dell’s involvement in and affinity for open source software
  • (5:31) Dell’s interest in and strategy around courting developers
  • (7:35) Dell’s strategy of Make, Partner or Buy in the cloud space
  • (11:10) How real is OpenStack and how is it evolving.

Extra-credit reading

Pau for now…


How to create a Basic or Advanced Crowbar build for Hadoop

November 29, 2011

As I mentioned in my previous entry, the code for the Hadoop barclamps is now available at our github repo.

To help you through the process, Crowbar lead architect Rob Hirschfeld has put together the two videos below.  The first, Crowbar Build (on cloud server), shows you how to use a cloud server to create a Crowbar ISO using the standard build process.  The second,  Advanced Crowbar Build (local) shows how to build a Crowbar v1.2 ISO using advanced techniques on a local desktop using a virtual machine.

Crowbar Build (on cloud server)

Advanced Crowbar Build (local)

Pau for now…


Developers: How to get involved with Crowbar for Hadoop

November 8, 2011

In the previous entry I mentioned that we have developed and will be opensourcing “barclamps” (modules that sit on top of Crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop.  All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters.

If you would like to get involved, check out this 1 min video from Rob Hirschfeld talking about how:

Look for the code on the Crowbar GitHub repo by the last week of November.

Extra-credit reading:

Pau for now…


Dell to opensource software to ease Hadoop install & management

November 8, 2011

It wouldn’t be surprising if you were surprised to learn that Dell is developing software.  To say that this is an area we haven’t been known for in the past would be an understatement.  While we may not pose a direct threat to Microsoft any time soon, we have been coding in a few focused areas.  One of those areas is cloud installation and management and is represented by our project Crowbar.  While Crowbar began life simply as a way to install Openstack on Dell hardware, it has expanded from there.

Today’s news is that we have developed and will be opensourcing “barclamps” (modules that sit on top of crowbar) for: Cloudera CDH/Enterprise, Zookeeper, Pig, Hbase, Flume and Sqoop.  All these modules will speed and ease the deployment, configuration and operation of Hadoop clusters.  But don’t take my word for it.  Take a listen to Crowbar’s architect Rob Hirschfeld as he explains Crowbar and today’s announcement:

Look for the code on Crowbar GitHub repo by the last week of November.  If you want to get involved, learn how.

Extra-credit reading:

Pau for now…


Crowbar: Where its been and where its going

October 24, 2011

Rob Hirschfeld, aka “Commander Crowbar,” recently posted a blog entry looking back at how Crowbar came to be, how its grown and where he hopes it will go from here.

What’s a Crowbar?

If you’re not familiar with Crowbar, its an open source software framework that began life as an installation tool to speed installation of OpenStack on Dell hardware.  The project incorporates the Opscode Chef Server tool and was originally created here at Dell by Rob and Greg Althaus.  Just four short months ago at OSCON 2011 the project took a big step forward when, along with the announcement of our OpenStack solution, we announced that we were opensourcing it.

DevOps-ilicous

As Rob points out in his blog, as we were delivering Crowbar as an installer a collective light bulb went off and we realized the role that Chef and tools like it play in a larger movement taking place in many Web shops today: the movement of DevOps.

The DevOps approach to deployment builds up systems in a layered model rather than using packaged images…Crowbar’s use of a DevOps layered deployment model provides flexibility for BOTH modularized and integrated cloud deployments.

On beyond installation and OpenStack

As the team began working more with Crowbar, it occurred to them that its use could be expanded in two ways: it could be used to do more than installation and it could be expanded to work with projects beyond OpenStack.

As for functionality, Crowbar now not only installs and configures but once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

The first project beyond OpenStack that we used Crowbar on was Hadoop.  In order to expand Crowbar’s usage we created the concept of  “barclamps” which are in essence modules that sit on top of the basic Crowbar functionality.  After we created the Hadoop barclamp, others picked up the charge and VMware created a Cloud Foundry barclamp and DreamHost created a Ceph barclamp.

It takes a community

Crowbar development has recently been moved out into the open.  As Rob explains,

This change was reflected in our work on OpenStack Diablo (+ Keystone and Dashboard) with contributions by Opscode and Rackspace Cloud Builders.  Rather than work internally and push updates at milestones, we are now coding directly from the Crowbar repositories on Github.

So what are you waiting for?  Join our mailing list, download the code or ISO, create a barclamp, make your voice heard.  Who’s next?

Extra-credit reading:

Pau for now…


Check us out at next week’s OpenStack Design Summit

September 29, 2011

If you’re planning on attending the OpenStack Design summit and conference next week in Beantown you’ll have to check us out.  I’m bummed that I will be missing the  summit for the first time, I have a big internal presentation next week, but the rest of the Dell OpenStack crew will be there in force.  Dell is a sponsor at the event and we will have a keynote, speaking sessions and demos.

What have we got in the works?

Besides checking out Crowbar and our Openstack solution which we launched back at OSCON we will  have a whisper suite where we will be showing our latest and greatest stuff that is currently in the works.  If you’d like  to see what we have up our sleeve, email us at OpenStack@Dell.com and we can schedule a time slot for you to come and see for yourself.

Updated: For more details what we’ll be doing at the summit check out Rob’s blog

Extra-credit reading

Pau for now…


Props from GigaOm for Dell as Web outfitter

September 26, 2011

Dell has been working for the last four plus years outfitting the biggest of the big web superstars like Facebook and Microsoft Azure with infrastructure.   More recently we have been layering software  such as Hadoop, OpenStack and crowbar on top of  that infrastructure.  This has not gone unnoticed by web pub GigaOm:

Want to become the next Amazon Web Services or Facebook? Dell could have sold you the hardware all along, but now it has the software to make those servers and storage systems really hum.

They also made the following observation:

Because [Dell] doesn’t have a legacy [software] business to defend, it can blaze a completely new trail that has its trailhead where Oracle, IBM and HP leave off.

Letting customers focus on what matters most

Its a pretty exciting time to be at Dell as we continue to move up the stack outfitting web players big and small.  The idea is to get these players established and growing in an agile and elastic way so they can concentrate on serving customers rather than building out their underpinning software and systems.

Stay tuned for more!

Extra-credit reading

Pau for now…


Cloud Foundry picks up Crowbar to speed installation

August 17, 2011

In case you’re not familiar with Cloud Foundry, it’s an open source Platform as a Service project initiated at VMware.  More specifically it provides a platform for building, deploying, and running cloud apps using Spring for Java developers, Rails and Sinatra for Ruby developers, Node.js and other JVM frameworks including Grails.

The project began two years ago when VMware’s CEO Paul Maritz recruited Derek Collison and Mark Lucovsky out of Google and set them to working on Cloud Foundry.  Collison and Lucovsky, who built and maintained Google’s API services, were brought into leverage their experience of working with hugely scaled out architectures.

The Cloud Foundry project has only been public for a matter of months and one question that I’m sure has popped into your mind is what if I want to pilot Cloud Foundry in my own environment, won’t installation and configuration be a total pain?

Enter the Crowbar

Crowbar is an open source software framework developed at Dell to speed up the installation and configuration of open source cloud software onto bare metal systems.  By automating the process, Crowbar can reduce the time needed for installation from days to hours.

The software is modular in design so while the basic functionality is in Crowbar itself, “barclamps” sit on top of it to allow it work with a variety of projects.  The first use for crowbar was for OpenStack and the barclamp for that has been donated to the community.  Next came The Dell | Cloudera solution for Apache Hadoop and, just recently, Dreamhost announced that they currently working on a Ceph barclamp.  And now…

Two great tastes that taste great together

Today’s big news is that VMware is working with Dell to release and maintain a Crowbar barclamp that, in conjunction with Crowbar, will install and configure Cloud Foundry.  This capability, which will include multi-node configs over time, will allow organizations and service providers the ability to quickly and easily get pilots of Cloud Foundry up and running.

Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the instance, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

If you’d like to try out Crowbar for yourself, check out: https://github.com/DellCloudEdge

Press added after initial posting

Extra-credit reading

Pau for now…


Introducing the Dell | Cloudera solution for Apache Hadoop — Harnessing the power of big data

August 4, 2011

Data continues to grow at an exponential rate and no place is this more obvious than in the Web space.  Not only is the amount exploding but so is the form data’s taking whether that’s transactional, documents, IT/OT, images, audio, text, video etc.   Additionally much of this new data is unstructured/ semi-structured which traditional relational databases were not built to deal with.

Enter Hadoop, an Apache open source project which, when combined with Map Reduce allows the analysis of entire data sets, rather than sample sizes, of structured and unstructured data types.  Hadoop lets you chomp thru mountains of data faster and get to insights that drive business advantage quicker.   It can provide near “real-time” data analytics for click-stream data, location data, logs, rich data, marketing analytics, image processing, social media association, text processing etc.  More specifically, Hadoop is particularly suited for applications such as:

  • Search Quality — search attempts vs. structured data analysis; pattern recognition
  • Recommendation engine — batch processing; filtering and prediction (ie use information to predict what similar users like)
  • Ad-targeting – batch processing; linear scalability
  • Thread analysis for spam fighting and detecting click fraud —  batch processing of huge datasets; pattern recognition
  • Data “sandbox” – “dump” all data in Hadoop; batch processing (ie analysis, filtering, aggregations etc); pattern recognition

The Dell | Cloudera solution

Although Hadoop is a very powerful tool, it can be a bit daunting to implement and use.  This fact wasn’t lost on the founders of Cloudera who set up the company to make Hadoop easier to used by packaging it and offering support.   Dell has joined with this Hadoop pioneer to provide the industry’s first complete Hadoop Solution (aptly named “the Dell | Cloudera solution for Apache Hadoop”).

The solution is comprised of Cloudera’s distribution of Hadoop, running on optimized Dell PowerEdge C2100 servers with Dell PowerConnect 6248 switch, delivered with joint service and support. Dell offers two flavors of this big data solution: Cloudera’s distribution with the free download of Hadoop software, and Cloudera’s enterprise version of Hadoop that comes with a charge.

It comes with its own “crowbar” and DIY option

The Dell | Cloudera solution for Apache Hadoop also comes with Crowbar, the recently open-sourced Dell-developed software, which provides the necessary tools and automation to manage the complete lifecycle of Hadoop environments.  Crowbar manages the Hadoop deployment from the initial server boot to the configuration of the main Hadoop components allowing users to complete bare metal deployment of multi-node Hadoop environments in a matter of hours, as opposed to days. Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect a complete data analytics solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

The solution also comes with a reference architecture and deployment guide, so you can assemble it yourself, or Dell can build and deploy the solution for you, including rack and stack, delivery and implementation.

Some of the coverage (added Aug 12)

Extra-credit reading

 

Pau for now…


Dell announces availability of OpenStack solution; Open sources “Crowbar” software framework

July 26, 2011

Today at OSCON we are announcing the availability of the Dell OpenStack Cloud Solution along with the open sourcing of the code behind our Crowbar software framework.

The Solution

Dell has been a part of the OpenStack community since day one a little over a year ago and today’s news represents the first available cloud solution based on the OpenStack platform.  This Infrastructure-as-a-service solution includes a reference architecture based on Dell PowerEdge C servers, OpenStack open source software, the Dell-developed Crowbar software and services from Dell and Rackspace Cloud Builders.

Crowbar, keeping things short and sweet

Bringing up a cloud can be no mean feat, as a result a couple of our guys began working on a software framework that could be used to quickly (typically before coffee break!) bring up a multi-node OpenStack cloud on bare metal.   That framework became Crowbar.  What Crowbar does is manage the OpenStack deployment from the initial server boot to the configuration of the primary OpenStack components, allowing users to complete bare metal deployment of multi-node OpenStack clouds in a matter of hours (or even minutes) instead of days.

Once the initial deployment is complete, Crowbar can be used to maintain, expand, and architect the complete solution, including BIOS configuration, network discovery, status monitoring, performance data gathering, and alerting.

Code to the Community

As mentioned above, today Dell has released Crowbar to the community as open source code (you can get access to it the project’s GitHub site).  The idea is allow  users to build functionality to address their specific system needs.  Additionally we are working with the community to submit Crowbar as a core project in the OpenStack initiative.

Included in the Crowbar code contribution is the barclamp list, UI and remote API’s, automated testing scripts, build scripts, switch discovery, open source Chef server.  We are currently working with our legal team to determine how to release the BIOS and RAID which leverage third party components.  In the meantime since it is free (as in beer) software, although Dell cannot distribute it, users can directly go the vendors and download the components for free to get that functionality.

More Crowbar detail

For those who want some more detail, here are some bullets I’ve grabbed from Rob “Mr. Crowbar” Hirschfeld’s blog:

Important notes:

  • Crowbar uses Chef as it’s database and relies on cookbooks for node deployments
  • Crowbar has a modular architecture so individual components can be removed, extended, and added. These components are known individually as “barclamps.”
  • Each barclamp has it’s own Chef configuration, UI subcomponent, deployment configuration, and documentation.

On the roadmap:

  • Hadoop support
  • Additional operating system support
  • Barclamp version repository
  • Network configuration
  • We’d like suggestions!  Please comment on Rob’s blog!

Extra-credit reading

Pau for now…


Structure: Learning about DevOps & Crowbar from Jesse Robbins

June 27, 2011

Last week on Day two of Structure the morning sessions ended with an  interesting discussion moderated by James Urquhart.  The session was entitled “DevOps – Reinventing the Developers Role in the Cloud Age” and featured Luke Kanies – CEO, Puppet Labs and Jesse Robbins – Co-Founder and CEO, Opscode.

After lunch I ran into Jesse and got him to sit down with me and provide some more insight into DevOps as well as explain what Opscode was doing with project Crowbar.

Some of the ground Jesse covers

  • (0:21) What is DevOps
  • (1:00) The shift that happens between developers and operations.  Writing code and getting it into production faster and how it shifts responsibilities between the two groups.
  • (2:52) Who are the prime targets for DevOps and how has this changed over time.
  • How DevOps began in web shops who needed to do things differently than legacy-bound enterprises.
  • How enterprises faced with greenfield opportunities are now embracing devops
  • (5:36) The crowbar installer which employs Opscode’s Chef and allows the rapid provisioning of an OpenStack cloud.

Extra-credit reading:


Citrix to launch OpenStack Distro (with a little help from Rackspace and Dell)

May 25, 2011

Today at Citrix Synergy, Citrix announced “Project Olympus,” their up coming OpenStack distribution.  In case you’re not familiar with it, OpenStack is an open source cloud platform based on the code from NASA’s Nebula cloud as well as Rackspace’s storage code.  The OpenStack project kicked of last summer and already has gathered support from over 60 commercial hardware and software vendors.

Mt. Olympus and the Cloud

Citrix’s OpenStack Distro

Citrix’s Project Olympus will produce a commercial distribution of the OpenStack infrastructure-as-a-service platform.  This “Olympian” distribution will be made up of two main components: a Citrix-certified version of OpenStack and a cloud-optimized version of XenServer.  While Citrix will lead with their Xen technology, thanks to OpenStack the distro will support all leading hypervisors.

Project Olympus is targeted at both public cloud providers as well as enterprise customers looking to build out private clouds.  The distribution will be available later this year.

But I want it now — The Citrix/Rackspace/Dell Early Access Program

For those who don’t want to wait until the official distribution is ready, don’t fret you can get started today through the Early Access Program (EAP).  The EAP is designed to help customers kick-off pilots and proof-of-concept deployments.  The program provides access to a beta version of the Citrix distro plus Dell hardware and deployment software as well as deployment services, training and on going customer support for customer clouds via Rackspace’s Cloud Builders program.

To get going with Citrix’s Project Olympus Early Access Program, please visit  http://www.citrix.com/olympus

Dell, Crowbar and Reference Architectures

Dell’s above-mentioned deployment software, aka “crowbar,” was a big hit at the last OpenStack Design summit.  The software which leverages Opscode’s Chef, allows folks to get an Openstack cloud up in running in less than four hours (instead of  days).  In addition to the deployment software and systems, to support the project Olympus EAP, Dell will also be providing reference architectures so keep your eyes peeled for those.

If you have any questions about what Dell is doing with OpenStack or want to get started, email us at OpenStack@Dell.com.

Extra-credit reading

Pau for now…


Follow

Get every new post delivered to your Inbox.

Join 88 other followers

%d bloggers like this: