Database Continuum On The Cloud - From Schemaless To Full-Schema

A recent paper by Mike Stonebraker and others compared relational and columnar database in a parallel configuration with MapReduce. The paper concludes that MapReduce is an easy to configure and easy to use option where as the other data stores, relational and columnar databases, pay the upfront price of organizing the data but outperform MapReduce in the runtime performance. This study does highlight the fact that a chosen option does not necessarily dictate or limit the scale as long as the other attributes such as an effective parallelism algorithm, B-tree indices, main-memory computation, compression etc. can help achieve the desired scale.

The real issue, which is not being addressed, is that even if the chosen approach does not limit the scale it still significantly impacts the design-time decisions that developers and architects have to make. These upfront decisions limit the functionality of the applications built on these data store and reduces the overall design-agility of the system. Let's look at the brief history of the evolution of DBMS, a data mining renaissance, and what we really need to design a data store that makes sense from the consumption and not the production view point.

Brief history of evolution of DBMS

Traditionally the relational database systems were designed to meet the needs of transactional applications such as ERP, SCM, CRM etc. also known as OLTP. These database systems provided row-store, indexes that work for selective queries, and high transactional throughput.

Then came the BI age that required accessing all the rows but fewer columns and had the need to apply mathematical functions such as aggregation, average etc. on the data that was being queried. Relational DBMS did not seem to be the right choice but the vendors figured out creative ways to use the same relational DBMS for the BI systems.

As the popularity of the BI systems and the volume of data grew two kinds of solutions emerged - one that still used the relational DBMS but accelerated the performance via innovative schema and specialized hardware and the other kind, columnar database, that used column-store instead of row-store. A columnar DBMS stores data grouped in columns so that a typical BI query can read all the rows but fewer columns in single read operation. Columnar vendors also started adding compression and main-memory computation to accelerate the runtime performance. The overall runtime performance of BI systems certainly got better.

Both the approaches, row-based and columnar, still required ETL - a process to extract data out of the transactional systems, apply some transformation functions, and load data into a separate BI store. They did not solve the issue of "design latency" - upfront time consumed to design a BI report due to the required transformation and a series of complicated steps to model a report.

Companies such as Greenplum and Aster Data decided to solve some of these legacy issues. Greenplum provides design-time agility by adopting a dump-all-your-data approach to apply the transformation on the fly only when needed. Aster Data has three layers to address the query, load, and execute aspects of the data. These are certainly better approaches that uses the parallelism really well and has cloud-like behavior but are still designed to patch up the legacy issues and do not provide clean design-time data abstraction.

What do we really need?

MapReduce is powerful since it is extremely simple to use. It has only three functions - map, split, and reduce. Such schemaless approaches have lately grown popularity due to the fact that developers don't want to lock themselves into a specific data model. They also want to explore adhoc computing before optimizing the performance. There are also extreme scenarios such as FriendFeed using relational database MySQL to store schema-less data. MapReduce has very low barrier to entry to get started. On the other hand a fully-defined schema approach by relational and columnar DBMS offers great runtime performance once the data is loaded and indexed for transactional access and executing BI functions such as aggregation, average, mean etc.

What we really need is a continuum from a schemaless to a full schema database based on the context, action, and access patterns of the data. A declarative abstracted persistence layer to access and manipulate the database that is optimized locally for various actions and access patterns is the right approach. This will allow the developers to fetch and manipulate the data independent of the storage and access mechanism. For example, developers can design an application where single page can perform a complex structured and unstructured search, create a traditional transaction, and display rich analytics information from single logical data store without worrying about what algorithms are being used to fetch and store data and how the system is designed to scale. This might require a hybrid data store architecture that optimizes the physical storage of data for certain access patterns and uses redundant storage replicated in real-time and other mechanisms such as accelerators for other patterns to provide unified data access to the applications upstream.

Schemaless databases such as SimpleDB, CouchDB, and Dovetail are in their infancy but the cloud makes it a good platform to support the key requirements of schemaless databases - incremental provisioning and progressive structure. Cloud also makes it a great platform for the full-schema DBMS by offering utility-style incremental computing to accelerate the runtime performance. A continuum on the cloud may not be that far-fetched after all.

Amazon's Re-designed Review System Generates More Revenue But Has Plenty Of Untapped Potential

Amazon's design tweaks to its review system has resulted into $2.7 billion of new revenue argues Jared Spool. Other people have also picked up this story with their analysis. I am wary of absolute revenue numbers tied to a feature to derive lost opportunity cost since a variety of other things could have driven the sale. It is wrong to assume that people would not have bought the products had the feature not existed. However I do believe it is a great step in the direction of making the review system more useful and drive more clickthroughs and conversions. Simply the presence of the reviews, magic number 20 in this case, motivates consumers to drill down into the details of a product and its reviews.

Amazon has made significant progress in collaborative filtering through their review system and it is an exemplary of a long tail business model. It has helped consumers to gain transparency and has also helped expose issues with the products. This is not enough. As an e-commerce market leader I would want Amazon to continue innovating around their review system. This is what I specifically would like to see in Amazon's review system:

Mining social media channels: Amazon.com is not the only place where consumers talk about the products. Consumers discuss product features and frustrations on Facebook, Twitter, and other social media outlets. Amazon has an opportunity to provide unified product review experience, a tool similar to ConvoTrack, by tapping into these social media channels for all the product conversations.

Tag cloud as a visual filter: One of the ways to make sense out of large number of reviews is to generate a tag cloud from the raw text of the reviews. A tag cloud acts as a great visual filter to narrow down the reviews that the consumers are looking for e.g looking only at rebooting issues and not anything else while buying a router.

Provide diverse search options: I want to search for the routers that have 4 or 5 stars ratings in the last 6 months. I cannot do that today. This search criteria makes sense. Manufacturers fix defects via firmware updates and models tend to improve as they mature. If the item had many negative reviews early on there is no way to find out without reading the other positive reviews whether the issues have been fixed or not. Higher recent ratings tend to correlate with mature product and satisfied customers.

Re-think one-size-fits-all format: All the products sold on Amazon ranging from a book to a TV has the exact same review format. It does not have to be that way. The book reviews tend to be more subjective and philosophical where the gadget reviews are generally more fact-based e.g watch out this monitor does not come with a DVI cable. Re-thinking the format for the types of products being sold make sense e.g pros and cons section for the gadgets, similar books to the one that I am reviewing etc.

Incentivise people to write reviews: Few days after consumers receive a product ask them whether they are satisfied with their purchase or not. Incentivize them to write reviews on the product; not only this helps generating more reviews per product but it also brings people back to Amazon to make more purchases. Make promotional email personal and relevant e.g.

How are you liking the "Tipping Point"? Malcom Gladwell has authored his latest book called "Outliers" and we are positive you will enjoy that as well. Would you mind writing a brief review of "Tipping Point" and we will discount the Outliers for you by 5%.

Closed-loop feedback channel: The current comments structure does not allow the manufacturers, authors, and the publishers to identify themselves and clarify the features, issues, and respond to consumers' concerns. The reviews are a great platform and a closed-loop feedback channel for the vendors to converse with the consumers. Amazon could certainly extend the review system to help create a dialogue between the consumers and the manufacturers.

Accelerating Social Computing: Web 2.0 + Cloud = Web²

I was at the Web 2.0 expo in San Francisco last week. It was not very different from the previous year except that I could see the impact of slow economy - shrinking attendance, less crowded booths, and "Hire Me" ribbons. Tim O'Reilly's keynote was interesting. He said that Web 2.0 was never about the version number (read, he does not like people calling Web 3.0 a successor of Web 2.0). He had the equation Web 2.0 + World = Web Squared. I changed it to Web 2.0 + Cloud = Web². The cloud seems more appropriate and the superscript is much cooler. If this catches on, remember, you read it here first!



The biggest shift that I have observed in Web 2.0 is the exponential growth of social media. This was evident at the Web 2.0 expo by looking at the number of participating social computing companies. Web 2.0 is certainly taking the direction of social computing. Tim mentioned in his keynote that the immense data gathered by the sensors and other means have hidden meaning in it and the applications have begun to understand this meaning. I could not agree any more; that justifies replacing the word "World" with "Cloud". Amazon's recent announcement to offer MapReduce on EC2 and Cloudera's $5M series A funding to commercialize Hadoop are early indicators of the rising demand for data-centric massive parallel processing. The cloud is a natural enabler to this evolution that will help gather data, context, and the interactions to amplify social conversations and create network effects. As John Maeda in his keynote said - people want to be human again. As Bill Buxton says :

User-centered design commonly tries to take into account different canonical user types through the use of persona. Perhaps one thing we need to do is to augment this tool with the notion of "placona," that is, capturing the canonical set of physical and social spaces within which any activity we are trying to support might be situated. After all, cognition does not reside exclusively in the brain. Rather, it is also distributed in the space in which we exercise that knowledge—in the location itself, the tools, devices, and materials that we use, and the people and social context in which all of this exists.

If one of the purposes of design and innovation is to improve our lives—for business, artistic, or familial purposes—then design that does not consider the larger social, cultural, and physical ecosystem is going to miss the mark.

Social computing is fundamentally a distributed problem that requires to make sense out of people's social and physical interactions with other people and objects including the context. The cloud can make this feasible and we can truly accelerate towards Web². I think Tim will most likely drop the 2.0 from Web 2.0 next year - that in itself would be a great first step in leaving Web 2.0 behind and start the journey towards Web².

Design Thinking Sustainability

The designers have been designing tools, processes, and methods to support and not to change people's behavior. In contrast designing for sustainability would fundamentally require changing people's behavior. The behavior change to achieve the sustainability goals would mean offering different alternatives, encourage reduced consumption, make people conscious of their behavior, and leverage peer pressure and competition. The design that maintains status quo will not help to achieve sustainability goals. The design will have to be provocative and challenge user's assumptions in many ways.

Design Thinking is about how you think and not what you know; it is about the journey and not the destination. For a problem of massive scale such as sustainability where we still know a little and the desired outcome may take years, following are some elements of design thinking that could help make world a better place to live for the generations to come.

Ambidextrous-thinking: Sustainability being fundamentally a sociological, psychological, and economical problem designers not only need to synthesize what they observe but to also design their solutions based on well-analyzed hard facts. It requires the designers to use both sides of their brains, left and right, to feel and to think. Human beings respond to positive and negative incentives e.g. charging people for grocery bags, allow hybrids cars in the carpool lanes etc. Ambidextrous approach allows designing creative incentives such as showing real-time gas consumption in Prius that changes the driver's behavior. It also prevents blindly rolling out initiatives that feel right but are outright wrong such as paper bags instead of plastic. Even though they are easy to down-cycle paper bags consume more energy to manufacture compared to plastic bags. All types of reasoning - inductive, deductive, and abductive - are quintessential to dream, design, and validate the solutions.

Analogous research: This approach allows designers to explore analogous problems with similar characteristics in other domains to gain insights and be inspired. Weight loss programs and alcohol support groups use social levers such as community support, peer pressure, and competition to help change people's behavior. The green social networks such as Carbonrally and Climate Culture are designed to leverage social competition towards green living. Similarly the community support aspect behind the fast growing fitness chain for women, Curve, can be applied to understand the role of community in changing people's behavior. Wiser Earth is an effort in this direction that uses community to connect people with non-profit and businesses to work together towards a sustainable world.

Researching an analogous domain is even more important when the primary domain such as sustainability does not allow to experiment the solution effectively due to its dry, non-tangible, and emerging nature. When given a task to design an emergency room a few people from IDEO went to a NASCAR race to observe the pit crew to better understand what kind of things can go wrong under emergency and how people respond to those events.

Empathy: Put yourself in the shoes of the people you are trying to change. As Thomas Friedman says people are having a green party and not a green revolution. Go to these green parties and follow people around to better understand what it will take to turn these parties into a true green revolution. Is it lack of awareness, motivation, or an incentive? Gain empathy for the people and understand their perspective in their context - what will it take socially and economically for them to change their behavior?

Context is critical for design thinking. Observing and talking to people in their natural environment designers gain empathy for the people and discover behavior patterns that they would have not found had they sat in their offices thinking how they should change people's behavior.

Holistic multidisciplinary approach: The sustainability efforts span across different culture, countries, background, and belief systems. To successfully solve this problem from the tools, behavior, and policies perspective people from the different disciplines such as engineers, scientists, interaction designers, social scientists, policy makers, and business executives need to come together and collaboratively work on it. Naive, curious, and inclusive mindset allows designers to holistically study the problem from the perspective of all the stakeholders - manufacturers, consumers, policy makers etc. The tools, technology, incentives, and policies, if designed in isolation, leave out gaps and often result into confirmation bias.

Be tangible and iterate often: This is a daunting problem and boiling the ocean would lead to an analysis paralysis nightmare. This is not a mature domain and there are no certainties around what will work and what won't. The best approach would be to rapidly prototype a solution to get early feedback from the consumers and iterate it often. There has been an ongoing debate on carbon tax versus carbon cap-and-trade. Instead of getting stuck in the controversy, opinions, and abstract ideas there is an opportunity to build something tangible and let people validate their own assumptions. A tangible object against an abstract concept enables better conversations and feedback channels since it is about the solution and not about the problem.

Focus on journey and emergent experimentation: Design thinking is about thinking in a different way and not about having any specific skills. It focuses on the journey, the method, and not on the outcome. People demand instant gratification but sustainability is not like the biggest looser competition where your weekly weigh-in would tell you where you are. It will take us years before we can actually quantify the impact of sustainability efforts that we are asking people to put in today. It is one of those initiatives that may not see any short term benefits at all. For such initiatives top-down compliance strategy won't work. A good design with emergent experimentation will focus on the journey and not the destination with the iterative results on the way to convince people how a change in their behavior slowly change the world around them. People will believe in the journey and the emergent experimentation.

Update: John R. Ehrenfeld who is currently serving as an Executive Director of the International Society for Industrial Ecology prior to his career as the Director of the MIT Program on Technology, Business, and Environment, an interdisciplinary educational, research, and policy program has picked up this story and posted on his blog Sustainability by Design.

Cloud Computing Strategy Crucial For eBay To Become One-stop E-commerce Shop

"We were the biggest and the best. And when you're the biggest and the best, there's a strong tendency to try to preserve that.....EBay has a storied past. But frankly, it's a past we've held onto too much." This is what the CEO of eBay, John Donahoe, told the analysts while explaining eBay's three-year revival plan to achieve low single digit growth. The outlined plan calls for continued investment into Skype, PayPal, and secondary-channels with an end goal to move eBay beyond an auction place to make it a one-stop e-commerce shop. Lack of explicit diversification plans to capture the fast growing e-commerce market (15% to 20% in the next five years) makes me wonder if eBay correctly assesses its core and context at this juncture and has the right infrastructure to support its strategy.

eBay's application-led multi-generation platform strategy has supported its core business strategy really well but eBay is significantly under-invested in the cloud computing to meet the new challenges. Unified e-commerce experience requires connecting heterogeneous and radically disparate data sources, applications, and their capabilities to monetize the traffic across them. Even if eBay does not tightly integrate Skype with PayPal and auction it should have the infrastructure to mine the information from Skype to support its strategy to gain major share of the e-commerce market especially to go after the digital goods such as e-books, songs, videos, ring tones etc. It would be a step backwards to limit the strategy view to the physical goods and a narrow transactional platform and not look at the holistic total customer experience that is SaaS delivered with the help of dynamic and elastic cloud-based platform.

MySpace is ahead in the game to monetize the immense clickstream data with the help of the cloud computing. eBay has huge untapped value in the social interaction and e-commerce data that it collects from all its assets. A right analytic cloud platform could make eBay rich from this gold mine. Social media cloud computing strategy is crucial for eBay not only to make competition irrelevant by going after emerging secondary markets but also to better prepare eBay for the megatrends such as millennial and sustainability. If eBay does successfully execute this strategy, tomorrow's eBay may not look like what you and I have seen so far.

Top Cloud, Virtualization, and SaaS Blogs - My Blog Makes The Cut

The organizers of the Under The Radar Conference has listed me as "best bloggers and journalists who’ve distilled this foggy space down to a well-defined, understandable sector" in the category of the top cloud, virtualization, and SaaS blogs. Some of the other names include Nick Carr, Geva Perry, and MR Rangaswami. I have also been invited to the conference. I am looking forward to the event and the energy that all the early stage start-ups bring in!

Creating, Nurturing, and Sustaining Innovation Clusters

McKinsey has partnered with the World Economic Forum to create an “Innovation Heat Map" to identify innovation clusters based on the analysis of the variables that drive innovation. The clusters are plotted on a classic McKinsey 2x2 that measures the size of a cluster on momentum versus diversity. I would encourage you to read the detailed analysis and look at the innovation heat map. The clusters are classified as:

Dynamic oceans: Large and vibrant innovation ecosystems with continuous creation and destruction of new businesses. Leading innovators and primary sectors change organically as the hub frequently reinvents itself through significant breakthrough innovations.

Silent lakes: Slow-growing innovation ecosystems backed by a narrow range of very large established companies that operate in a handful of sectors. These clusters are frequently the source of a steady stream of “evolutionary” innovations and step-wise improvements.

Shrinking pools: Innovation hubs that are unable to broaden their areas of activity or increase their lists of innovators and so find themselves slowly migrating down the value chain, as their narrow sector becomes less innovation driven and increasingly commoditized.

Hot springs: A small and fast-growing hub that relies on a small number of companies to establish itself as a relevant world player in a narrow sector.

While I applaud the efforts behind analyzing the vast amount of indicators to find patterns that explain certain macroeconomic innovation trends I disagree with the idea of measuring innovation based on number of patents. The effectiveness of our (US) patent system to measure innovation effectively across industries is questionable. How many of these patents are actually converted into real innovations? This problem is exasperated when we involve the patent systems of the countries across the globe.

This analysis assumes minimum infrastructure base as a qualifier to level the playing field. Though I do appreciate the intent of this assumption, this approach leaves out the countries who innovate despite of poor infrastructure such as India and China. An alternate approach could have been a weighted cluster that focuses on the efficiency of the clusters to demonstrate the untapped innovation potential due to lack of infrastructure.

These clusters provide an opportunity to explore some correlations. Hot Springs regions' growth is correlated to untapped natural resources and recent influx of skilled immigrants. This might explain Canada and Australia being Hot Spring regions due to their untapped natural resources and the immigration policies designed to attract highly skilled an well educated prospective immigrants.

To make this analysis even more compelling I would like to go back few years more than the current nine years of data and choose time-based visualization such as Gap Minder. This might reveal some interesting patterns about how the clusters grow from small to big and vice versa and change quadrants as the years go by. This visualization would not only allow to add filters but would also allow to track relative progress of a subset of clusters. We might also be to see how the clusters move from silent lake to dead pool since they cannot innovate themselves out of the current crisis e.g. auto and manufacturing regions in the US.