The Future Of BI In The Cloud

Actual numbers vary based on whom you ask, but the general consensus is that the Business Intelligence (BI) and Analytics in the cloud is a fast growing market. IDC expects a compounded annual growth rate (CAGR) of 22.4% through 2013. This growth is primarily driven by two kinds of SaaS applications. The first kind is a purpose-specific analytics-driven application for business processes such as financial planning, cost optimization, inventory analysis etc. The second kind is a self-service horizontal analytics application/tool that allows the customers and ISVs to analyze data and create, embed, and share analysis and visualizations.

The category that is still nascent and would require significant work is the traditional general-purpose BI on large data warehouses (DW) in the cloud. For the most enterprises, not only all the DW are on-premise, but the majority of the business systems that feed data into these DW are on-premise as well. If these enterprises were to adopt BI in the cloud, it would mean moving all the data, warehouses, and the associated processes such as ETL in the cloud. But then, the biggest opportunities to innovate in the cloud exist to innovate the outside of it. I see significant potential to build black-box appliance style systems that sit on-premise and encapsulate the on-premise complexity – ETL, lifecycle management, and integration - in moving the data to the cloud.

Assuming that the enterprises succeed in moving data to the cloud, I see a couple of challenges, if treated as opportunities, will spur the most BI innovation in the cloud.

Traditional OLAP data warehouses don’t translate well into the cloud:

The majority of on-premise data warehouses run on some flavor of a relational or a columnar database. The most BI tools use SQL to access data from these DW. These databases are not inherently designed to run natively on the cloud. On top of that, the optimizations performed on these DW such as sharding, indices, compression etc. don’t translate well into the cloud either since cloud is a horizontally elastic scale-out platform and not a vertically integrated, scale-up, system.

The organizations are rethinking their persistence as well as access languages and algorithms options, while moving their data to the cloud. Recently, Netflix started moving their systems into the cloud. It’s not a BI system, but it has the similar characteristics such as high volume of read-only data, a few index-based look-ups etc. The new system uses S3 and SimpleDB instead of Oracle (on-premise). During this transition, Netflix picked availability over consistency. Eventual consistency is certainly an option that BI vendors should consider in the cloud. I have also started seeing DW in the cloud that uses HDFS, Dynamo, and Cassandra. Not all the relational and columnar DW systems will translate well into NoSQL, but I cannot overemphasize the importance of re-evaluating persistence store and access options when you decide to move your data into the cloud.

Hive, a DW infrastructure built on top of Hadoop, is a MapReduce meet SQL approach. Facebook has a 15 petabytes of data in their DW running Hive to support their BI needs. There are a very few companies that would require such a scale, but the best thing about this approach is that you can grow linearly, technologically as well as economically.

The cloud does not make it a good platform for I/O intensive applications such as BI:


One of the major issues with the large data warehouses is, well, the data itself. Any kind of complex query typically involves an intensive I/O computation. But, the I/O virtualization on the cloud, simply does not work for large data sets. The remote I/O, due to its latency, is not a viable option. The block I/O is a popular approach for I/O intensive applications. Amazon EC2 does have block I/O for each instance, but it obviously can’t hold all the data and it’s still a disk-based approach.

For BI in the cloud to be successful, what we really need is ability for scale-out block I/O, just like scale-out computing. Good news is that there is at least one company, Solidfire, that I know, working on it. I met Dave, the founder, at the Structure conference reception. He explained to me what he is up to. Solidfire has a software solution that uses solid state drives (SSD) as scale-out block I/O. I see huge potential in how this can be used for BI applications.

When you put all the pieces together, it makes sense. The data is distributed across the cloud on a number of SSDs that is available to the processors as block I/O. You run some flavor of NoSQL to store and access this data that leverages modern algorithms and more importantly horizontally elastic cloud platform. What you get is commodity and blazingly fast BI at a fraction of cost with pay-as-you-go subscription model.

Now, that’s what I call the future of BI in the cloud.

Can A Product Manager Be Effective Without Product Design Skills?

I am very passionate about the topic of design and design-thinking. When I saw this question on Quora, I decided to post my answer. Following is directly from my answer to this question on Quora:

The answer is "Definitely not."

It's not about the product design by itself, but it's about applying core and transferable product design skills to product management. Let's break it down:

1) Understanding users: Good product designers have great user research, observation, and listening skills to put themselves into the shoes of a user and understand the real, mostly unspoken and latent, needs of the end users.

2) Being self-critical: If you are a trained designer, you would stay away from self-referential design, which is a root cause for many failed products. Good product designers are self-critical about their approach and the deliverables and are always open to feedback to iterate on their design.

3) Working with designers: If you are a designer, you have great empathy for fellow designers. I have seen products fail, simply because, the product managers can't work with the designers and don't share the same mindset.

4) A "maker" mentality: The designers are makers. They make things. The product managers typically don't, the engineers do. For a product manager, it's incredibly important to have a "maker" mentality. They should continuously be making and refining, by themselves or with the help of the engineers. The product managers, who believe that their responsibility ends when they are done gathering the requirements are likely to fail, miserably in most cases.

5) A "T-shaped" product manager: If you're a product manager, the vertical line of the "T" is your core PM skills. However, successful product managers go beyond their core skills, the horizontal line in the letter "T", to learn more about product design, engineering etc. This ensures that they have a holistic perspective of the product. That leads me to my last point.

6) General Manager: viable, feasible, and desirable: A good product from a vendor's perspective is commercially viable, technologically feasible, and desirable by the end users. Many product managers stop at the business needs, but they truly need to go beyond that to work with the engineering to make it technologically feasible, and have a design mindset to work with the designers to make it desirable by the end users. The product managers should thrive for a "general manager" mindset, of which, product design is a core element.

Telcos Could Be The Future Enterprise Software Vendors For Small Businesses

Having worked on enterprise software product and go-to-market strategy for SMB (small and medium businesses), I can tell you that these are the most difficult customers to reach to, especially the S in SMB. It’s an asymmetric non-homogeneous market for which the cost of sales could go out of control if you don’t leverage the right channels. The competitive landscape varies from region to region and industry to industry. In many cases instead of competing against a company you would be competing against a human being with paper-based processes.

Tomorrow I am speaking at the Razorsight annual conference on the topic of cloud computing. I am excited to meet their customers, the telcos. While I prepare for my keynote, I can’t stop thinking about the challenges that the telcos face and the opportunities that they are not pursuing. My keynote presentation is about how telcos can leverage the cloud, but this blog post is about how telcos can become successful enterprise software vendors and market their solutions to small businesses.

There are very few things that are common across small businesses. They own a landline (at least for now) and they have Internet access, in many cases from the same vendor. I believe that the landlines will be more and more difficult to sell to these customers, but losing a channel – a relationship – would be even worse. If leveraged well, these relationships could be worth a lot more compared to the landline business as it stands today. Just think about it. Selling to small businesses is all about leveraging existing relationships with them. This channel is priceless.

What will it take for the telcos to market products to small businesses?

ISV acquisitions or VAR agreements: If telcos are bundling software, on-premise or SaaS, the telcos, as organizations, don’t necessarily have the skills or resources to make software for small businesses. This would mean a series of small and niche ISV acquisitions across geographical areas and industries and VAR agreements with current ISVs.

What kind of software can telcos bundle?

There are two kinds: horizontal and vertical. The examples of horizontal software are accounting, payroll, point of sale etc. Ask Intuit and they will tell you all about the horizontal cash cow. The vertical software is industry specific for the business that you are in. One of my favorite companies in this area is OpenTable. If you have made an online reservation at a restaurant you have most likely used their software. They had a successful IPO last year and they are on track to become a $100 million company.

Telcos should be doing all these things. They have cash and they can borrow cheap money to buy companies. Telcos also have an option to leverage the cloud, their own cloud in many cases, to provide SaaS solutions to small businesses. They can leap frog the on-premise ISVs who don’t have access to these customers and are sensitive to margin cannibalization.

Lean Startup Customer Development And IxD Personas

On Quora Steve Blank asked "Is it possible to use Lean Startup customer development findings to inform IxD personas?" This post is my response to Steve on Quora:

Absolutely yes.

Pivoting is not just about finding the right business model that works for a start-up but it is also about nailing down the persona that you are designing your product for. I have seen many start-up fail because they don't know who the end user is. Creating a persona is an iterative process by itself. Many people focus on persona as a final artifact but I believe that the journey is more important than the destination. While discovering a persona and iterate on it to make it crisp, the team - the dev, marketing, and UX - comes together with the shared understanding of the target end user. The journey brings in the empathy that they all internalize and that influences what they do. The journey includes getting out of the office and talk to the real people who you think would use your product.

Persona requires qualitative discovery as well as validation. It's an instantiation of your customer. The customer discovery, validation, and creation are all directly related to the persona. In fact I would argue that in many cases knowing the target audience, at a given stage, is far more important than having a perfect product. Plenty of people fixate on building the right product against building it for the right people.

A Laundromat Entrepreneur

In my previous post “While Entrepreneurs Scale On The Cloud The Angels Get Supersized” I wrote about how cloud computing is disrupting the VC industry. Continuing on the thread of entrepreneurship I am seeing more and more entrepreneurs building applications who do not belong to any formal organization, start-up or otherwise. The definition of what used to be a start-up itself is changing, primarily because of two reasons - simple and easily accessible PaaS tools to design, run, and maintain applications on the cloud and access to a market place to sell the applications.

We have been witnessing this trend for the mobile applications for a while - Android as well as iPhone and now iPad. I see the same pattern for the cloud-based applications. I have seen many useful, productive, and successful applications that are designed by individual developers with no affiliation to any organization.

Google has done a great job in designing the tools for the developers to build applications that can run on their cloud and can be sold on their app store. This has democratized the application business to large extent that attempt to solve niche problems. At the same time the individual developers have started monetizing their work without going through an overhead of bootstrapping and running a company. While Google’s cloud platform is a generic one the application and stack specific PaaS providers such as Salesforce.com and Heroku are also attracting such developers. Intuit’s partner development platform is a great example of a channel platform that allows the entrepreneurs to market to an SMB segment, a very difficult segment to reach (a post on that later).

All these trends, collectively, have introduced a new category of an entrepreneur. A laundromat entrepreneur.

They are not full fledged start-ups but these individuals are also not developing just for fun. These businesses have steady revenue, positive cash flow, and require very little maintenance. The companies such as Help Me - located in Karachi, Pakistan - have created their business model to support such developers outsource customer support for their existing applications so that they can focus on building new applications. Some of these individual businesses could be worth a few million dollars.

This is a very different business model that combines the best-of-breed with long tail. I am quite excited about this new category since that puts in the developers directly in charge of the product and takes them closer to the end users. I am curious to see the life cycle of these laundromats and how they get bought and sold. Many people that I have had discussions with claim that we could expect to see plenty of individuals who will own such a laundromat portfolio worth five to six million dollars.

Attribution: I have shamelessly stolen the word “laundromat” from my friend Mike Ni after my discussion with him on cloud computing business models. I had told him that I would!

The picture credit to Michael Valli

While Entrepreneurs Scale On The Cloud The Angels Get Supersized

Cloud computing is disrupting the venture capital industry in a big way. One of the obvious changes we all have observed is the reduced up-front capital expenditure to start a new venture. Things that used to require an array of expensive servers and an army of people to maintain them have essentially been replaced by a bunch of EC2 instances and a few smart developers. The tools and the technology stack for today’s applications are designed for cheaper and faster experimentation allowing the entrepreneurs to follow the lean methodology and pivot as fast as they can. I agree that some investors underestimate the people cost and overestimate the capabilities of the cloud but regardless this has caused a major shift in how the companies are funded.

The rise of an emergent category of super angel is all about leveraging the cloud computing. Fred Wilson closed a $30 million fund and Aydin Senkut closed a $40 million fund. These funds will invest into dozens of companies that can be bootstrapped with low up-front cost. More and more entrepreneurs prefer to raise as little money as possible in the beginning. This phenomenon has a few effects:

Raise AS you scale and not raise TO scale:

Founders have been able to raise money at good valuation without giving up large equity. This has been an uneasy situation for many venture capitalists and has crated strange problems while raising money. When Foursquare raised money the founders sold part of their equity to the VCs so that the VCs can earn money on a successful exit. The founders also decided not to sell out to Yahoo. Raising money as the company scales follows the cloud motto of scale-as-you-need and pay-as-you-go.

Build a product that you want and not what a VC wants:

The super angels typically stay on the sidelines and definitely don’t serve on the board. This means a lot more freedom to entrepreneurs to define and shape their product. This also allows the companies to take up-front risk, venture into new areas, and experiment where conventional wisdom would otherwise have prevented them. Fail fast and fail cheap is now a reality from a venture as well as technology perspective.

Prominent network effects in the start-up community:

I strongly believe that the cloud is the best participatory platform to create network effects of all kinds. I have seen similar kind of network effects occur in the new angel industry, especially in an incubator such as Y Combinator. The Silicon Valley start-ups have enjoyed the network effects for long time. These effects are even more profound when some of these start-ups are in an incubator setting. Such environments have a natural advantage for the entrepreneurs to leverage cross-pollination. Cloudkick is such an example of a YC company that was started by three entrepreneurs to build a solution to manage the Amazon EC2 instances that all other YC companies used at that time.

Competition in the portfolio companies could be a good thing:

The VCs do not prefer to have competing start-ups in single portfolio to avoid conflict of interest. As rational as it sounds this is simply not feasible when an angel or a super angel funds tens and hundreds of companies. I believe that it’s a good thing. At macro level the angels can see the patterns and advise the companies and at the micro level the companies can hone in their competitive differentiation before raising more money. This might also change how the founders pick and choose the angels. If the founders pick an angel who has similar companies in their portfolio they can expect better connections and mentoring from the angels despite of having the competing companies funded by the same set of investors.

It’s not that the entire VC industry has changed. The series A and B investors are as important as angels and super angels but the way the VCs operate and the expectations that the limited partners have would certainly change. I also believe that the VCs who are not stage agnostic will revisit their seed-funding strategy. The performance of the traditional VC funds that were raised in the last ten years is far worse than what an investor would expect from an alternate class assets, which is what the VC investments are. Time will tell whether doing more deals with same money will yield better return on the portfolio but, at least for now, the VC climate change is imminent.

Software Is The New Hardware

Today Intel announced that it is buying McAfee for $7.7 billion. This acquisition made people scratch their heads. Why McAfee?

The obvious arguments are that Intel has hit the growth wall and organic growth is not good enough to satisfy the shareholders. But this argument quickly falls apart from margin perspective. Why dilute their current nice gross margin even if McAfee has steady revenue stream? [Read my update at the end of the post]

I believe there are two reasons. The first is that the companies need a balanced product and revenue mix regardless of different margins. Oracle bought Sun and HP bought EDS. Big companies do this all the time. The second, not so obvious, reason is a recognition that software is new hardware. The processors are processors – they are a commodity any which way you look at them. It is not news to anyone that the computing has become commodity which is the basis of utility style cloud computing. Software, embedded or otherwise, has significant potential to sell value-added computing. The security solutions could fit in nicely on a piece of chip. When you drive a few miles from Intel’s headquarters to meet folks at nVidia you will be amazed to see what kind of value a software tool kit can derive from the processors.

I don’t know how Intel will execute the merger considering the fact that this is their largest acquisition ever. But, I am even more convinced that software is the new hardware. Cloud computing, data center automation, virtualization, network security, and a range of other technologies can leverage software in a chip that is optimized for a set of specialized tasks. Time to move from commodity to specialized computing until specialized computing becomes commodity. Interesting times!

Update: Romit sent me a message commenting that how McAfee will dilute Intel’s margin since McAfee’s gross margin is more than Intel. I should clarify. The assumption on the street is that the cost of capital for this purchase is about 4% and Intel expects 8% return on the investment even after paying 60% premium for the purchase. The tricky part is that how long Intel can maintain the close to 75% software margin of a software company operating inside a hardware company. When I say diluting the margin I mean diluting the overall combined margin post-purchase. The analysts are skeptical about the success of the merger and so am I. Intel has no track record of integrating large software companies such as McAfee especially after paying significantly higher than average premium. Hypothetically if Intel were to buy a company with more synergies that can leverage existing channels and can fit into their culture they could have increased the gross margin and hence the return to their shareholders.