Thursday, February 23, 2012

Simple Workflow Service - Amazon Adding One Enterprise Brick At Time



Yesterday, Amazon announced a new orchestration service called Simple Workflow Service. I would encourage you to read the announcement on Werner's blog where he explains the need, rationale, and architecture. The people I spoke to had mixed reactions. One set of people described this as a great idea and were excited that the developers can now focus on writing domain-specific code as opposed to writing plumbing code to orchestrate their actual code. The other set of people felt that this service creates a new cloud lock-in making it difficult for the developers to switch from one cloud to another as well as being able to interoperate at the orchestration level.

I believe this is a brilliant idea for a variety of reasons. Orchestration has always been painful. Ask the developers who have been involved in managing task execution across a cluster that required them to code for load balancing, handling exceptions, restarting hung processes, tracking progress etc. This is not a core competency the most developers have but they do end up writing such code due to lack of better alternative. The frameworks such as WS-BPEL were not designed to run in cloud-like environments and there has been no single standard REST orchestration framework out there that people could use.

From a vendor's perspective, I admire Amazon's ability to keep innovating via such services that differentiate them as a leading cloud vendor. As computing becomes more and more commodity, competing based on price alone isn't a good idea. If you're a cloud vendor you need to go above and beyond the traditional IaaS attributes even though you excel in all of them. I also see PaaS eventually bleeding into IaaS as IaaS continues to become a commodity. As far as PaaS goes, federated or otherwise, we're barely scratching the surface.

I don't see this service as a cloud lock-in but it certainly makes EC2 more attractive and sticky. I would be concerned if Amazon were to force the developers to use their SWS for orchestration. This is their version of how they think orchestration should be done and the developers can opt in if they want. And kudos to them to think beyond their cloud. The folks who worry about cloud lock-ins also talk about Amazon not following the standards. I believe that we should not create standards for the sake of creating standards. I am a believer in first showing that something works and later, if there's enough interest, figure out a way to standardize it. All these talks about standard-first even before you write that first line of code doesn't make any sense.

It's yet to be seen how this service turns out, but this is a huge step forward for getting more enterprise software customers onboard. Orchestration is one of the most chronic problems of enterprise software and with the challenges of a hybrid landscape to be able to orchestrate across on-premise and cloud-based solutions, this service is certainly a step in the right direction. Right Scale has been using a Ruby workflow Ruote for their workflow needs and now they orchestrate these workflows using SWS  to achieve fault tolerance and concurrency. As you can see, Amazon has opened up a gold mine for start-ups. The back-end execution has always been challenging. Now, there is an opportunity to write your own enterprise grade workflow engine or scheduler that runs in the cloud.

Friday, February 17, 2012

Wrong Side Of The IT Ecosystem



I find it ridiculous that people are blaming Apple for job creation in China as opposed to in the US. People are also debating how US might in-source some of these manufacturing jobs to compete with China who has sophisticated manufacturing abilities and large skilled labor force supporting these operations. They are all missing the point. This is a wrong debate.

The US lost manufacturing jobs to other countries a long time ago. I find it amusing that people expect the high-tech companies such as Apple to create manufacturing jobs in the US. If Apple were to even consider this option we would not have seen the tremendous success of Apple as a company and its products. What Apple created is an ecosystem of people and companies that are doing amazing things with their platform and their devices. It's a different kind of ecosystem and America should focus on that innovation as opposed to bringing those manufacturing jobs back.

On one side we are whining about the loss of manufacturing jobs and on the other side we have shortage of skilled IT workforce. Try hiring a good developer in the Silicon Valley and you'll understand what I mean. And yet as a nation we are behind in retraining our existing workforce, attracting students to engineering majors, and fixing our immigration policy for highly skilled foreign workers to meet the increased demand of IT-driven jobs. And, of course, while we wait, Apple is quadrupling its IT investment in India.

America should not play the manufacturing game with China or for that matter with anyone else. We are at such a loss. Let's play the game that we know we can win — technology-driven innovation. When I work with customers' on daily basis I come across so many opportunities that we are not looking at. We can use the technology, that we have built, to our advantage in the industries such as healthcare, agriculture, public sector etc. A combination of cloud and mobility could take us long way.

We're looking at the wrong side of the IT ecosystem. I don't expect the hi-tech companies to hire low-tech workers in the US. But I do expect hi-tech companies to create jobs in the US at the other end of the ecosystem via the opportunities to consume their technology and innovate in a different sector. A lot of people are missing this point. I'm talking about an ecosystem where Apple has paid out more than $4 billion to the developers. Why are we not talking about these jobs? Apple has more than $100 billion in cash, but what doesn't get much discussed is that a large part of this cash is overseas. Given the current US tax laws, Apple can't/won't bring this cash back into the US. This might make Apple acquiring or investing overseas. We do have an opportunity to reform the tax laws to deal with such a global situation (that we never encountered before) to encourage the hi-tech companies to invest into R&D in the US and not overseas.

When you look at the big picture, having a job is merely one piece of contributing to good standards of living. What about access to affordable healthcare and college education? There's a significant opportunity to apply technology built in America to innovate in these areas. We are barely scratching the surface of what's possible in healthcare as well as in education. We are living in such an antiquated and inefficient system.

Another industry that has seen less than desired technology innovation is agriculture. Take a trip down to central California to see the potential. At 2008 Olympics in China, Michael Phelps winning 8 gold medals was obviously the biggest highlight for us, but people in Watsonville were super excited because China allowed the US to export Watsonville strawberries for the Olymipcs. Recently, India relaxed the laws (that are still being challenged) to allow 100% foreign investment in the retail sector opening up the doors for Wallmarts of the world. Any guess what's the biggest challenge in retail operations in India? A non-existent cold supply chain and lack of reliable infrastructure. We take a lot of things for granted — nationwide freeways, strong governing bodies such as FDA, and size of the country. We do have an opportunity to excel in certain agriculture areas and employ a lot of Americans. We need to recognize what our real strength is and look forward as opposed to look backwards.

I am a geek and a technology enthusiast, and perhaps a little naive. But, I know for sure, we aren't pushing the technology envelope as much as we should.

Photo: Looking Towards Tomorrow

Tuesday, January 31, 2012

To RIM: Don't Change The Strategy, Change The Rules


A lot has been said and discussed about RIM's downfall: indecisive leadership, inability to innovate at fast pace, and no clear path to recovery. I don't disagree at all with the analysis and the interpretation of the situation, but I do disagree with the conclusion that many people are drawing and vehemently disagree with their advice to RIM to keep trying to regain the smartphone market share. That train has left the station and RIM doesn't have a chance to catch up, even if they do everything that they could.

But RIM may have stumbled upon something that they probably least expected. It's the BlackBerry Messaging, popularly known as BBM. We got to see the power of BBM during the London riots. During my recent trip to India, I firsthand witnessed how much of people's lives depend on BBM. These people were sad, upset, and depressed due to a RIM infrastructure outage. This is a phenomenal success. The recipe behind this success is quite simple: provide free messaging that looks likes SMS that supports groups in a network. RIM has significantly leveraged network effects; BBM got better as more and more people used it. The sale of BlackBerry in India has gone viral. The consumers buy Blackberries since their friends have it so that they can chat with them for free and perhaps do their emails. These consumers don't use any apps at all! Their needs are quite simple. These phones are also priced well - the median price is somewhat around $200 for an unlocked phone. The Indian middle class and upper middle class have no issues shelling out this money to buy a BlackBerry. I talked to quite a few people and they are moving away from Android and iPhone to BlackBerry. Yes, that's right. If RIM can manage to introduce lower end versions of BlackBerry this will further fuel the growth.

May be, just may be, there's a category between smart and non-smart phones. For a large number of people in emerging economies making a phone call and staying in touch with their friends and family via text messages and email, and not paying too much for doing that are the driving reasons to purchase a right kind of a phone.

Let's briefly look at the history of RIM. It was the device of choice for email and calendering and perhaps still is for a lot of people. RIM myopically focused on going after the enterprise customers while iPhone and Android pulled the rug underneath them. RIM initially ignored and later underestimated the disruptive nature of this innovation. What started out as a consumer market, iPhone and Android easily crossed the chasm and entered into an enterprise and started replacing BlackBerry. We all know this story. But, something happened during this era of RIM: they ended up building a massively scalable and reliable enterprise class messaging infrastructure. This is an amazing feat of technical excellence. Building BlackBerry Messaging was a logical extension of leveraging this infrastructure. What if RIM uses this as a strength and not worry about competing in the smart OS area.

It's time to pivot.

Build a robust phone that is primarily driven off by BlackBerry Messaging and double down in emerging economies. Change the rules of the game and beat Nokia at its own strategy. Even better, spin off BlackBerry into two separate businesses: one that exclusively focuses on this strength and the other that embraces innovation by OEMing either Android or Windows or both and defend the handset as well as the services market share. I don't believe BlackBerry is cut out to innovate on a new smart phone OS quick enough to beat iOS or Android or an emerging contender, Windows phone. That would mean playing by your competitors' rules. If you learn one thing from Apple, it would be not to do this.

I don't need to tell you how many cellphones the Indians own and how many of those can buy a BlackBerry. This may not be an intended move, but this social effects driven business in emerging economies such as India as well as in other developed countries could be the second act for BlackBerry. Can other vendors replicate this? May be. Not many companies in the world can do what BlackBerry does with emails and messaging in general. Group messaging on a mobile device is a killer app in itself to drive the sales of handsets. Also, this works across the carriers and the geographies, essentially allowing RIM not to be threatened by a provider. SMS GupShup in India has been an extremely popular group messaging service. It's a validation that there is significant untapped potential for RIM.

Photo courtesy: NoHoDamon

Thursday, January 19, 2012

Subscribe To Own As New Lease To Own



The Beatles are timeless and so is music and enterprise software.

There's been an ongoing innovation in the music services. iTunes with iPods disrupted the traditional CD business model and in the ever connected cloud world Pandora, Spotify and countless others are challenging the very concept of "owning" music. Spotify gives you access to a wide range of music on all their clients as long as you're a paid subscriber. This is like Netlfix model for music except that there's no ad-supported free Netflix (Spotify is rumored to cap the free version after six months of usage). Pandora also has a similar model but it's a "radio" service. You can't tell Pandora what exactly to play but give preferences and it will find, play, and tune music based on your preference.

Pandora is serendipitous and Spotify is spontaneous.

One of the challenges with these services is that you only have access to music as long as you pay for it. When you stop using it you don't own anything (from them). iTunes and Amazon, on the other hand, are a music "marketplace". You buy songs and keep them. But these services are not designed for you to explore and experiment endlessly.

I wonder whether there's a middle ground.

What if there's a subscribe to own business model? The services would stream all the music that you want for a fixed price (like Spotify) and users will get a choice of receiving certain number of DRM-free songs — like options being vested, at the end of the subscription plan — say annually. The studios may never agree to this, but it's a great value proposition. What if a service is designed to actually sell MP3s and the streaming is just a draw to get people discover new music? Also, imagine if Netflix were to give out credit to their streaming customers to own a set of movies on DVDs. "Lease to own" is a very popular way of buying a car (at least in the United States). Why not apply that to music?

It is very difficult to change human behavior. The studios are powerful, want full control, and see technology innovation as a threat as opposed to an opportunity. On the other side, consumers are willing to pay and experiment but they do want to own music so that they can play on any device any which way they want without getting locked into a specific service and its supported clients.

What about SaaS subscription models for enterprise software? Are there any issues when customers don't "own" the software that they are using?

I have blogged about SaaS escrow and inverted OEM channels before. We haven't yet seen any spectacular failures of large SaaS companies. Today, even if you're a large unprofitable SaaS vendor with a decent customer base, you will be acquired before you shut your doors for good. But once SaaS becomes the de facto mode of delivering software, the "hotness" will fade away and you are as likely to go out of business as any other ISV. What happens then? The customers have their business continuity plans and a SaaS vendor going out of business could become a serious concern.

As far as music goes, there's a clear separation between content and process. We listen to music which is content and everything else — streaming, matching, discovering, and recommending — is a process to get to the content. This clear separation is not that clear in enterprise software. SaaS escrow could guarantee the content (of course if vendor supports it) but not the process and without process there's not much of business continuity. You could take your music and go some place else but I doubt you can do much with your enterprise data without any process around it. Is there an analogous flavor of lease to own in enterprise SaaS business? I guess, it's too early to say.

Going back to music, I think, we're ready for a radical shift and disruption in existing business models. Lease to own your music may not be a bad idea after all.

Friday, December 30, 2011

Loving What I Do For Living



A few months back, I was helping a very large customer of ours to help simplify as well as automate their process of trading financial instruments. During one of my many visits to their office, I met a person who was trying to explain to me his job in supporting the people that are involved in this super complex process. I always ask a lot of questions — until they're totally annoyed and ready to kick me out of the room — to get a complete understanding of the business rationale behind whatever they're thriving for and their personal motivation behind it. Something unusual happened at this meeting. Instead of getting into the gory technical details of how they get things done, he chose to tell me a short and simple story.

"You know, um.. there's this early morning meeting everyday that Peter goes to with a bunch of other people. They all gather around a large table in a dimly lit conference room with a bunch of printed spreadsheets, a laptop, and a large calculator. Peter has a cup of coffee in one hand and a cigarette in the other hand talking to people who have coffee cups in their one hand and cigarettes in the other hand. This is their lives. I am concerned about Peter and I want him to stop smoking. Can you please help me?"

Now, this is the job that I love that makes me get out the bed and run for it. This is the human side of enterprise software. It's not boring.

Photo Courtesy: Jane Rahman

Wednesday, December 14, 2011

Design thinking: A New Approach To Fight Complexity And Failure


Photo credit: String Theory by Michael Krigsman

The endless succession of failed projects forces one to question why success is elusive, with an extraordinary number of projects tangling themselves in knots. These projects are like a child’s string game run amok: a large, tangled mess that becomes more convoluted and complex by the minute.

IT projects fail all the time. Business blames IT, IT blames the system integrator (SI), who then blames the software vendor. After all this blaming and shaming, everyone goes back to work on another project without examining the project management methods and processes that caused the failure. And, so, they fail again.

There’s no one definition of design thinking. It’s a mindset and set of values that applies both analytical and creative thinking towards solving a specific problem. Design thinking is about how you think and not what you know; it is about the journey and not the destination.

Having followed Michael Krigsman’s analysis of IT project failures, it became evident that design thinking can play an important role in improving enterprise software development and implementation. 
The design thinking approach offers a means to address the underlying causes of many project failures — poor communication, rigid thinking, propensity toward tunnel vision, and information silos.

I have distilled important lessons from design thinking into six principles that can help stop project failures. Along the way, we will draw comparisons with Agile development, since that distinction is often a source of confusion when discussing design thinking.

These six principles, based on design thinking, can help any project team operate more successfully.

1. Put a multi-disciplinary team in charge

You can’t pin down project failure on one person or one topic and yet we continue to use a person-centric method to manage projects. No one on a project team wants to fail. If you collectively put responsibility of the failure or success on the shoulders of the team and get them trained and motivated to think and behave differently you will mitigate much failure.

Multidisciplinary teams champion the user, business, and technology aspects of a project in a more comprehensive manner than would otherwise be possible. Typically, an IT team talks to business stakeholders who then talk to end users, which creates communication gaps, delays, and inefficiency. Far better to create a single team that includes participants from all areas, creating a single unit that includes multiple perspectives.

Try to staff your project team with “T-shaped” people, who possess a broad understanding and empathy for all the IT functions, but who also have deep expertise in one domain to champion that perspective. This approach can ensure that your solution is economically viable, technologically feasible, and delights the end users. A more balanced team also humanizes the project and its approach. Stay small and resist the temptation to set up very large teams. If you believe the “two-large-pizza-team” rule, those projects are team-driven and tend to be more successful. Start-ups can build something quicker because they are always short on people. As your group get bigger and bigger, other people tell you what to do and team members feel less connected to their work as it relates to the outcome.

2. Prepare for failure in the beginning

I recommend kicking off the project with a “pre-mortem workshop.” Visualize all the things that could go wrong by imagining that the project has failed. This gives the team an opportunity to proactively look at risks and prepare to prevent and mitigate them. I have sat through numerous post-mortem workshops and concluded that the root causes of failures are usually the same: abstract concepts such as lack of communication, unrealistic scope, insufficient training, and so on. If that’s true, why do we repeat the same mistakes, causing failure to remain a common situation? Primarily because many people find it hard to imagine and react to abstractions, but can relate much better when these concepts are contextualized into their own situation.

3. Be both vision- and task-driven

Design thinking emphasizes storytelling, shared vision, and empathy towards all stakeholders involved in a project. On many projects, participants focus exclusively on their own individual tasks, thus becoming disconnected from the big picture.

While design thinking strives to connect participants to the larger vision, Agile development can be very task-driven. Everyone gets a task without necessarily understanding the big picture, or vision, or even seeing the connection between his or her tasks and the final outcome. In this situation, a project can fail and people may not understand their role, thinking they failed due to someone else’s work. If participants don’t realize their tasks contributed to a failure, they won’t try to learn and change.

On the other hand, vision-driven approaches are very powerful. People perform their tasks, but the story and vision persist throughout the project; the same story gets told by different people throughout the lifecycle of the project to avoid that big picture fading away. All the tasks have a bigger purpose beyond their successful execution. Even good project managers miss this point. At review meetings, it is important to evaluate what the team did right but also revisit the vision and examine how recent outcomes fit the overall story.

4. Fail and correct then fail again

Design thinking contradicts other methodologies that focus only on success. In design thinking, failing is not necessarily a bad idea at all; however, we fail early and fail often, and then correct the course. In many projects, people chase success without knowing what it looks like or expecting to fail; therefore, they do not learn from the process.

One of the challenges with traditional project management is the need to pick one alternate and run with it. Turns out that you don’t know everything about that alternative and when it fails, due to the irreversible decision that you made, you can’t go back. Far better to iterate on a number of alternatives as fast as you can before deciding which one will work. This approach requires a different way of thinking and planning your project.

5. Make tangible prototypes

Agile proposed creating unstructured documentation as opposed to making structured requirement documents. But, unfortunately, that is not enough to solve many problems. One of the core characteristics of design thinking is to prototype everything, to make a tangible artifact and learn from it. The explorative process of making prototypes makes people think deeply and ask the right kind of questions. It’s said that “computers will never give a wrong answer but it will respond to a wrong question.” The prototypes encourage people to focus on what I want to know as opposed to what I want to say. This is very important during the initial design phase of the project.

One of the biggest misconceptions about prototypes is that people think they are too complex to make and are overhead or a waste of time. This isn’t true at all. Prototypes can be as simple as a hand-drawn sketch on a paper or as complex as fully functional interactive interface. The fidelity of a prototype is based on what kind of questions you want answered. People tend to fill in gaps when they see something raw or incomplete whereas hi-fidelity prototypes can be too complete to solicit meaningful feedback. As I already mentioned, most people respond better to an artifact as opposed to an abstract document. Prototypes also make the conversation product-centric and not person-centric. They also help to get team members on the same page with a shared vision.

6. Embrace ambiguity

One of the problems with traditional project management methodologies is that they make people spend more time in executing the solution and less time on defining the problem. Design thinking encourages people to stay in the problem space as long as they can. This invariably results in ambiguity, which is actually a good thing.

Ambiguity fosters abductive thinking — a mindset that allows people to explore what is probable with the limited information on their hands without concerns about proving or concluding that it actually works. It helps people define a problem in many different ways, eventually letting them get to the right problem they eventually should focus on.

This also supports the emergent approach that design thinking advocates as opposed to a hypothesis-driven approach. In a hypothesis-driven environment, people tend to focus on proving a premise created by a small group people. Rushing to a solution without defining the problem, and having no emergent framework in place to include the insights gained during later parts of the project, certainly contributes to failure.

ORGANIZATIONAL BARRIERS TO SUCCESS

Even the best methodology requires organizational commitment to success. For design thinking to work, it is also necessary to address these common organizational issues, each of which can impede progress and limit successful outcomes.

Lack of C-level commitment: Although design thinking is applicable at all levels in an organization, executive management must bless it by publicly embracing and practicing design thinking. Top down initiatives and training only go so far.

When the employees see their leaders practice design thinking they are more likely to embrace and practice it themselves. The same is true with adoption of social media and collaborative tools inside an organization. The best signal to your employees is by showing them a firm belief in the method by practicing it firsthand and sharing positive outcome.

Resistance to change: People in any organization are usually fundamentally against change, even if they believe it’s a good thing. They don’t want to get out of their comfort zone and therefore practice the same methods that have resulted in multiple failures in the past. Changing behavior is difficult but fortunately design thinking can help.

One of the ways I have taught design thinking is by taking people away from their primary domain and have them solve a very different kind of problem such as redesigning a ticket vending machine or a fast food restaurant. My team was hugely successful since it was a completely different domain and it didn’t interfere with their preconceived notion of how a project should be executed. People’s reservations are tied to their domain; they are willing to adopt a new method and new way of thinking if you coach them outside of their domain and then encourage to practice it in their comfort zone.

Lack of industry backing: Despite being informal, undocumented, and non-standards-based methodology, Agile experienced widespread adoption. I would attribute this success to two things: a well-defined manifesto by lead industry figures and organizations publicly committing to adopt the methodology. Design thinking lacks these attributes.

Even though industrial design companies such as IDEO has evangelized this approach, there’s still confusion around what design thinking actually means. This also makes it difficult to explain design thinking to a wider audience. If a few organizations publicly endorse design thinking, create a manifesto, and share the best practices to gain momentum, many of the adoption hurdles will go away.

Lack of key performance indicator (KPI) frameworks: Design thinking faces the same challenge that most Enterprise 2.0 tools face: lack of measurable KPIs.

For number-driven leaders, lack of a quantifiable framework to measure and monitor the impact of a new methodology is a challenge. Some leaders are good at adopting new ways of doing things and others are not. In these cases, isolate a project that you can’t measure and start small. Contain the risk but pick a project that has significant upside, to keep people engaged and motivated. You may still fail, or not achieve a desired outcome, but that’s what the design thinking is all about.

It’s worth noting that Agile, as a software project methodology, has well defined quality and reliability KPIs such as beta defects, rejected stories during a scrum cycle, and the delta between committed and delivered stories.

Fail early and course correct the next time. Remember that adoption and specific practice need correction and not the method itself. Don’t give up.

FINAL THOUGHTS

During my extensive work on design thinking - practicing, coaching, and analyzing — I often talk with people who believe that design thinking is merely a methodology or approach for “visual design.” This view is a false perception. Design thinking comprises a set of principles one can apply during any stage of the enterprise project lifecycle along with other project management methodologies. This approach is valid for the CEO and executive management all the way to the grass roots.

Another common point of confusion is the distinction between design thinking and Agile methods of software development. The primary difference is that Agile offers a specific set of prescriptive processes while design thinking encapsulates a set of guidelines and general principles. Although not the same, the two approaches are highly complementary (even on the same project), because both recognize the benefits of using iterative work cycles to pursue customer-centric goals.

Always remember that real people work on every project. The best methodologies are inherently people-centric and help participants anticipate likely causes of failure. Visualizing failure early in a project is an excellent means to prevent it from occurring. We’re all human and may make mistakes but certainly no one wants to fail.

Design thinking can make potential failure a learning tool and not a final outcome.
_______

I had originally published this post as a guest blog post on Michael Krigsman's IT Project Failures blog

Wednesday, November 30, 2011

Coming To A Place Near You: A Private Cloud Spiked With Big Data


Netflix similarity map
Yesterday, I moderated a couple of panels at the Big Data Cloud event. I have been a keynote speaker, panelist, moderator, and participant for many conferences in the last few years. It has always been a pleasure to see the cloud and big data becoming more and more mainstream. Here are my quick observations and insights from the event:

Private cloud getting momentum: As a public cloud proponent I thought I would never have to write this. But lately I have seen more and more interest in private cloud; new start-ups, established cloud vendors, and large legacy vendors are designing private or hybrid cloud solutions. Vendors have recognized that prospects and customers have started to take cloud very seriously but they still have the same concerns what they had few years back: security, moving data to public cloud, and giving up control. I am not interested in the private/public debate (though I do love to mess with fellow clouderati on Twitter on this topic). My take on this trend is that the vendors should do whatever it takes to move the organizations to the cloud, private or public. Once companies dip their toes, they themselves will realize what's good for them.

Big Data as a serious category: A few days back, I blogged about big data going mainstream. Coming out from this event, it felt like, today Big Data is where cloud was a couple of years back. When I asked people a few years back "What's Hadoop?" They would reply "Huh?" Now, everyone wants to know more about Hadoop, Hive, HBase, S4, Oozie, Pig, Cassandra, and other big data frameworks. They're interested in analyzing and comparing available solutions. They're asking all the right questions. The VC investment in this category has been record high. Hadoop World was a sold out event this year with 1500 participants. Milind Bhandarkar, Chief Architect with Greenplum Labs, mentioned that in 2008, during the first Hadoop summit, they had to coax people to come to the summit. The people who willingly came to the summit either worked for Yahoo or Facebook. We have come a long way and there's a long way to go but this is a rock solid category. As the first set of big data infrastructure companies settle in we will see people building killer applications and PaaS solutions specifically designed to leverage big data. It is encouraging to see more and more companies and venture capitalists recognizing that the data is worth a lot more if they have the right tools and right people — the data scientists — to do something interesting with it. For example, Greylock partners have hired DJ Patil as a "data scientist in residence" to help them with evaluating their opportunities and advising their portfolio companies on big data strategies.

Rise in popularity of open source frameworks: If you follow the history of open source you'll realize that when a proprietary way of doing things become popular, commercial vendors pose a lock-in threat, and things don't work as expected, developers get frustrated and start to work on filling that gap by building open source technology. Linux started that way and so were many other open source projects. This is why I am excited to see OpenStack gaining rapid momentum. It's slowly becoming a de-facto standard to build a commercial cloud solutions. I also like Cloud Foundry since many companies that I know of, ISVs and large IT shops, won't use a public PaaS. They would prefer to launch their own PaaS solution in the cloud. Without an open source solution, it does become a big challenge.