Proxies Are As Useful As Real Data

Last year I ran a highly unscientific experiment. I would regularly put a DVD in an open mail bin in my office to mail it back to Netflix, every late Monday afternoon. I would also count the total number of Netflix DVDs put inside that bin by other people. Over a period of time I observed a continuous and consistent decline in the number of DVDs. I compared my results with the numbers released by Netflix. They matched. I'm not surprised. Even though this was an unscientific experiment on a very small sample size with a high degree of variables, it still gave me insights into the overall real data, that I otherwise had no access to.

Proxies are as useful as real data.

When Uber decides to launch a service in a new city or when they are assessing demand in an existing city they use crime data as surrogate to measure neighborhood activity. This measurement is a basic input in calculating the demand. There are many scenarios and applications where access to a real dataset is either prohibitively expensive or impossible. But, a proxy is almost always available and it is good enough in many cases to make certain decisions that eventually can be validated by real data. This approach, even though simple, is ignored by many product managers and designers. Big Data is not necessarily solving the problem of access to a certain data set that you may need, to design your product or make decisions, but it is certainly opening up an opportunity that didn't exist before: ability to analyze proxy data and use algorithms to correlate them with your own domain.

As I have argued before, the data external to an organization is probably far more valuable than the data that they internally have. Until now the organizations barely had capabilities to analyze a subset of their all internal data. They could not even think of doing anything interesting with the external data. This is rapidly going to change as more and more organizations dip their toes in Big Data. Don't discriminate any data sources, internal or external.

Probably the most popular proxy is the per-capita GDP to measure the standard of living. The Hemline Index is yet another example where it is believed that the women's skirts become shorter (higher hemline) during good economic times and longer during not-so-good economic times.

Source: xkcd

Proxy is just a beginning of how you could correlate several data sources. But, be careful. As wise statisticians will tell you, correlation doesn't imply causation. One of my personal favorite example is the correlation between the Yankees winning the worldseries and a democratic president in the oval office. Correlation doesn't guarantee causation, but it gives you insights into where to begin, what question to ask next, and which dataset might hold a key to that answer.This iterative approach wasn't simply feasible before. By the time people got an answer to their first question, it was too late to ask the second question. Ability to go after any dataset anytime you want opens up a lot more opportunities. At the same time when Big Data tools, computing, and access to several external public data sources become a commodity it would come down to human intelligence prioritizing the right questions to ask. As Peter Skomoroch, a principal data scientist at LinkedIn, puts it "'Algorithmic Intuition' is going to be as important a skill as 'Product Sense' in the next decade."

I Want USPS To Think Outside The Box

Recently I had to go to a consulate to get a visa and the consulate would only accept a USPS money order and a USPS pre-paid envelope. I went to a post office to get those. That particular post office decided to change their business hours that day to open late. I hurriedly drove to a different post office where two out of there clerks didn't know how to issue a pre-paid envelope! At personal level I never look forward to going to a post office. It invariable delays my schedule. I am met with unpleasant customer service and inefficiency everywhere. This is also true with some of the other services that I get but there's one major difference. I cannot opt out of USPS.

USPS anticipates to lose about $7 billion during the fiscal year that ends in September. They even have their own conference called PostalVision 2020 where they have invited technology thought leaders such as Vint Cerf and many others to honestly and seriously look at the issues they have. The agenda is to:

"Postal Vision 2020/2.0 is as much a movement as it is a Conference.  It is a forum for an open and honest dialog to better understand the future of postal communications and shipping, and what this means to those who regulate, supply and use mail.  It’s about sharing ideas and knowledge with the hope of sparking innovation and the creation of new successful business models.  It’s about asking each other lots of difficult questions for which there may be many answers to consider before finding those that serve the long term health of the industry and any particular enterprise."

USPS is broken at so many levels; they have short term as well long term issues to deal with and it is likely to get uglier before it may get better. Channeling Geoffrey Moore, USPS needs to retain their core and and redefine the context. Massive fleet of trucks, logistics, and outlets in all foreseeable locations is their core strength. Postal mail and other related services is their context where they are simply unable to compete because of shrinking addressable market (due to digital communication) and poor service design that applies the legacy mindset to solve today's and tomorrow's problems.

USPS should think outside the box. No pun intended.  

Here are some ideas/suggestions:

Deliver groceries: Remember Webvan? I loved their service during the dot com boom. One of the main reasons they went out of business is they had no expertise on logistics. Since then nothing much has changed in home-delivered grocery business. What if USPS delivered grocery to your home? What if they partnered with a local supermarket and took over their logistics business? This is a complimentary business model. The supermarkets are not in the delivery business and it's not economical for them to enter into the logistics business. This is also a sustainable business that helps the environment. The USPS trucks are on the road no matter what, but now they can take a few cars off the road. This may sound crazy but times are changing and it's time for USPS to rethink what unfair advantage they have over others.

Re-think mail delivery: It's perfectly acceptable to me if I only receive my mail every other day. In many cases, I am fine if I don't get my mail for a week at times. There's nothing time-sensitive about my mail. And with changing demographics, this is true with a lot of other people as well. Incentivize customers to skip mail by offering them discount on other services and have less trucks and less people going around the neighborhoods. This brings the overall cost down and opens up new revenue opportunities.

Double down on self-service: I know USPS is trying hard to add more and more self-service kiosks but they're not enough. Think like Coinstar and Redbox. I should be able to do everything related to USPS at the places where I can get milk at the 11th hour, money from ATM, and gas for my car. They really need to work hard to give people a reason to use USPS when people have much better alternatives to mail packages. Think of UPS, DHL, and FedEx as incumbents and leap frog them at places, using the unfair advantage that USPS has, where they can't possibly compete.

Rethink the identity: USPS doesn't directly receive federal tax dollars and it is expected to meet expenses from the revenue it generates. But, it's not that black and white. Even though USPS doesn't get any tax money it receives plenty of other money via grants and other special funds. It's neither truly a government entity nor truly a business entity. If USPS needs to be fixed it needs to rethink its identity and decide whether it's a complete public sector or a mix of private and public sector and how. Once that identity is set they can follow through on their revenue sources, cost measures, and building an ecosystem of partners. Mixed and complicated business structure introduces complexity at all the levels and prevents the organization to think and execute in a unified way.

Data Is More Important Than Algorithms

Netflix Similarity Map

In 2006 Netflix offered to pay a million dollar, popularly known as the Netflix Prize, to whoever could help Netflix improve their recommendation system by at least 10%. A year later Korbel team won the Progress Prize by improving Netflix's recommendation system by 8.43%. They also gave the source code to Netflix of their 107 algorithms and 2000 hours of work. Netflix looked at these algorithms and decided to implement two main algorithms out of it to improve their recommendation system. Netflix did face some challenges but they managed to deploy these algorithms into their production system.

Two years later Netflix awarded the grand prize of $1 million to the work that involved hundreds of predictive models and algorithms. They evaluated these new methods and decided not to implement them. This is what they had to say:

"We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then."

This appears to be strange on the surface but when you examine the details it totally makes sense.

The cost to implement algorithms to achieve incremental improvement isn't simply justifiable. While the researchers worked hard on innovating the algorithms Netflix's business as well as their customers' behavior changed. Netflix saw more and more devices being used by their users to stream movies as opposed to get a DVD in mail. The main intent behind the million dollar prize for Netflix was to perfect their recommendation system for their DVD subscription plan since those subscribers carefully picked the DVDs recommended to them as it would take some time to receive those titles in mail. Customers wanted to make sure that they don't end up with lousy movies. Netflix didn't get any feedback regarding those titles until after their customers had viewed them and decided to share their ratings.

This customer behavior changed drastically when customers started following recommendations in realtime for their streaming subscription. They could instantaneously try out the recommended movies and if they didn't like them they tried something else. The barrier to get to the next movie that the customers might like significantly went down. Netflix also started to receive feedback in realtime while customers watched the movies. This was a big shift in user behavior and hence in recommendation system as customers moved from DVD to streaming.

What does this mean to the companies venturing into Big Data?

Algorithms are certainly important but they only provide incremental value on your existing business model. They are very difficult to innovate and way more expensive to implement. Netflix had a million dollar prize to attract the best talent, your organization probably doesn't. Your organization is also less likely to open up your private data into the public domain to discover new algorithms. I do encourage to be absolutely data-driven and do everything that you can to have data as your corporate strategy including hiring a data a scientist. But, most importantly, you should focus on your changing business — disruption and rapidly changing customer behavior — and data and not on algorithms. One of the promises of Big Data is to leave no data source behind. Your data is your business and your business is your data. Don't lose sight of it. Invest in technology and more importantly in people who have skills to stay on top of changing business models and unearth insights from data to strengthen and grow business. Algorithms are cool but the data is much cooler.

Fixing Software Patents, One Hack At Time

Software patents are broken and patent trolls are seriously hurting innovation. Companies are spending more money on buying patents to launch offensive strikes against other companies instead of competing by building great products. There are numerous patent horror stories I could outline where they are being used for all purposes except to innovate. In fact the software patent system as it stands today has nothing to do with innovation at all. This is the sad side of the Silicon Valley. While most people are whining about how software patent trolls are killing innovation some are trying to find creative ways to fix the problems. This is why it was refreshing to see Twitter announcing their policy on patents, Innovator's Patent Agreement, informally called IPA. As per IPA, patents can only be used in an offensive litigation if the employees who were granted the patents consent to it. I have no legal expertise to comment on how well IPA itself might hold up in a patent litigation but I am thrilled to see companies like Twitter stepping up to challenge the status quo by doing something different about it. If you're an employee you want three things: innovate, get credit for your innovation, and avoid your patents being used as an offensive tool. IPA is also likely to serve as a hiring magnet for great talent. Many other companies are likely to follow the suit. I also know of a couple of VCs that are aggressively pushing their portfolio companies to adopt IPA.

The other major challenge with software patents is the bogus patents granted based on obvious ideas. I really like the approach taken by Article One Partners to deal with such patent trolls. Article One Partners crowdsources the task of digging the prior art to identify bogus patents and subsequently forces the US patent office to invalidate them. Turns out that you don't have to be a lawyer to find prior art. Many amateurs who love to research this kind of stuff have jumped into this initiative and have managed to find prior art for many bogus patents. It's very hard to change the system but it's not too hard to find creative ways to fix parts of the system.

I would suggest going beyond the idea of crowdsourcing the task to find the prior art. We should build open tools to gather and catalog searchable prior art. If you have an idea just enter into that database and it becomes prior art. This would make it incredibly difficult for any company to patent an obvious idea since it would already be a prior art. We should create prior art instead of reactively research for it. Open source has taught us many things and it's such a vibrant community. I can't imagine the state of our industry without open source. Why can't we do the same for patents? I want to see Creative Commons of patents.

The industry should also create tools to reverse translate patents by taking all the legal language out of it to bring transparency to show for what purposes that patents are being granted for.

I would also want to see an open source like movement where a ridiculously large set of patents belong to one group - a GitHub of patents. And that group will go after anyone who attempt to impede innovation by launching an offensive strike. If you can't beat a troll then become one.

Silicon valley is a hacker community and hackers should do what they are good at, hack the system — to fix it — using creative ways.

Photo: Opensource.com

4 Big Data Myths - Part II

This is the second and the last part of this two-post series blog post on Big Data myths. If you haven't read the first part, check it out here.

Myth # 2: Big Data is an old wine in new bottle

I hear people say, "Oh, that Big Data, we used to call it BI." One of the main challenges with legacy BI has been that you pretty much have to know what you're looking for based on a limited set of data sources that are available to you. The so called "intelligence" is people going around gathering, cleaning, staging, and analyzing data to create pre-canned "reports and dashboards" to answer a few very specific narrow questions. By the time the question is answered its value has been diluted. These restrictions manifested from the fact that the computational power was still scarce and the industry lacked sophisticated frameworks and algorithms to actually make sense out of data. Traditional BI introduced redundancies at many levels such as staging, cubes etc. This in turn reduced the the actual data size available to analyze. On top of that there were no self-service tools to do anything meaningful with this data. IT has always been a gatekeeper and they were always resource-constrained. A lot of you can relate to this. If you asked the IT to analyze traditional clickstream data you became a laughing stroke.

What is different about Big Data is not only that there's no real need to throw away any kind of data, but the "enterprise data", which always got a VIP treatment in the old BI world while everyone else waited, has lost that elite status. In the world of Big Data, you don't know which data is valuable and which data is not until you actually look at it and do something about it. Every few years the industry reaches some sort of an inflection point. In this case, the inflection point is the combination of cheap computing — cloud as well as on-premise appliances — and emergence of several open computing data-centric software frameworks that can leverage this cheap computing.

Traditional BI is a symptom of all the hardware restrictions and legacy architecture unable to use relatively newer data frameworks such as Hadoop and plenty of others in the current landscape. Unfortunately, retrofitting existing technology stack may not be that easy if an organization truly wants to reap the benefits of Big Data. In many cases, buying some disruptive technology is nothing more than a line item in many CIOs' wish-list. I would urge them to think differently. This is not BI 2.0. This is not a BI at all as you have known it.


Myth # 1: Data scientist is a glorified data analyst

The role of a data scientist has exponentially grown in its popularity. Recently, DJ Patil, a data scientist in-residence at Greylock, was featured on Generation Flux by Fast Company. He is the kind of a guy you want on your team. I know of a quite a few companies that are unable to hire good data scientists despite of their willingness to offer above-market compensation. This is also a controversial role where people argue that a data scientist is just a glorified data analyst. This is not true. Data scientist is the human side of Big Data and it's real.

If you closely examine the skill set of people in the traditional BI ecosystem you'll recognize that they fall into two main categories: database experts and reporting experts. Either people specialize in complicated ETL processes, database schemas, vendor-specific data warehousing tools, SQL etc. or people specialize in reporting tools, working with the "business" and delivering dashboards, reports etc. This is a broad generalization, but you get the point. There are two challenges with this set-up: a) the people are hired based on vendor-specific skills such as database, reporting tools etc. b) they have a shallow mandate of getting things done with the restrictions that typically lead to silos and lack of a bigger picture.

The role of a data scientist is not to replace any existing BI people but to complement them. You could expect the data scientists to have the following skills:

• Deep understanding of data and data sources to explore and discover the patterns at which data is being generated. 
• Theoretical as well practical (tool) level understanding of advanced statistical algorithms and machine learning.
• Strategically connected with the business at all the levels to understand broader as well deeper business challenges and being able to translate them into designing experiments with data.  
• Design and instrument the environment and applications to generate and gather new data and establish an enterprise-wide data strategy since one of the promises of Big Data is to leave no data behind and not to have any silos.

I have seen some enterprises that have a few people with some of these skills but they are scattered around the company and typically lack high level visibility and an executive buy-in.

Whether data scientists should be domain experts or not is still being debated. I would strongly argue that the primary skill to look for while hiring a data scientist should be how they deal with data with great curiosity and asking a lot of whys and not what kind of data they are dealing with. In my opinion if you ask a domain expert to be a data expert, preconceived biases and assumptions — knowledge curse —  would hinder the discovery. Being naive and curious about a specific domain actually works better since they have no pre-conceived biases and they are open to look for insights in unusual places. Also, when they look at data in different domains it actually helps them to connect the dots and apply the insights gained in one domain to solve problems in a different domain.

No company would ever confess that their decisions are not based on hard facts derived from extensive data analysis and discovery. But, as I have often seen, most companies don't even know that many of their decisions could prove to be completely wrong had they have access to right data and insights. It's scary, but that's the truth. You don't know what you don't know. BI never had one human face that we all could point to. Now, in the new world of Big Data, we can. And it's called a data scientist.

Photo courtesy: Flickr

4 Big Data Myths - Part I

It was cloud then and it's Big Data now. Every time there's a new disruptive category it creates a lot of confusion. These categories are not well-defined. They just catch on. What hurts the most is the myths. This is the first part of my two-part series to debunk Big Data myths.

Myth # 4: Big Data is about big data

It's a clear misnomer. "Big Data" is a name that sticks but it's not just about big data. Defining a category just based on size of data appears to be quite primitive and rather silly. And, you could argue all day about what size of data qualifies as "big." But, the name sticks, and that counts. The insights could come from a very small dataset or a very large data set. Big Data is finally a promise not to discriminate any data, small or large.

It has been prohibitively expensive and almost technologically impossible to analyze large volumes of data. Not any more. Today, technology — commodity hardware and sophisticated software to leverage this hardware — changes the way people think about small and large data. It's a data continuum. Big Data is not just about technology, either. Technology is just an enabler. It has always been. If you think Big Data is about adopting new shiny technology, that's very limiting. Big Data is an amalgamation of a few trends - data growth of a magnitude or two, external data more valuable than internal data, and shift in computing business models. The companies mainly looked at their operational data, invested into expensive BI solutions, and treated those systems as gold. Very few in a company got very little value out of those systems.

Big Data is about redefining what data actually means to you. Examine the sources that you never cared to look at before, instrument your systems to generate the kind of data that are valuable to you and not to your software vendor. This is not about technology. This is about completely new way of doing business where data finally gets the driver's seat. The conversations about organizations' brands and their competitors' brands are happening in social media that they neither control nor have a good grasp of. At Uber, Bradly Voytek, a neuroscientist is looking at interesting ways to analyze real-time data to improve the way Uber does business. Recently, Target came under fire for using data to predict future needs of a shopper. Opportunities are in abundance.

Myth # 3: Big Data is for expert users    

The last mile of Big Data is the tools. As technology evolves the tools that allow people to interact with data have significantly improved, as well. Without these tools the data is worth nothing. The tools have evolved in all categories ranging from simple presentation charting frameworks to complex tools used for deep analysis. With rising popularity and adoption of HTML 5 and people's desire to consume data on tablets, the investment in presentation side of the tools have gone up. Popular javascript frameworks such as D3 have allowed people to do interesting things such as creating a personal annual report. Availability of a various datasets published by several public sector agencies in the US have also spurred some creative analysis by data geeks such as this interactive report that tracks money as people move to different parts of the country.

The other exciting trend has been the self-service reporting in the cloud and better abstraction tools on top of complex frameworks such as Hadoop. Without self-service tools most people will likely be cut off from the data chain even if they have access to data they want to analyze. I cannot overemphasize how important the tools are in the Big Data value chain. They make it an inclusive system where more people can participate in data discovery, exploration, and analysis. Unusual insights rarely come from experts; they invariably come from people who were always fascinated by data but analyzing data was never part of their day-to-day job. Big Data is about enabling these people to participate - all information accessible to all people.

Coming soon in the Part II: Myth # 2 and Myth # 1.

Learning From Elevators To Design Dynamic Systems

Elevators suck. They are not smart enough to know which floor you might want to go. They aren't designed to avoid crowding in single elevator. And they make people press buttons twice, once to call an elevator and then to let it know which floor you want to go to. This all changed during my recent trip to Brazil when I saw the newer kind of elevators.

These elevators have a common button panel outside in the lobby area of a high rise building. All people are required to enter their respective floor numbers and the machine will display a specific elevator number that they should get into. Once you enter into an elevator you don't press any numbers. In fact the elevators have no buttons at all. The elevator would highlight the floor numbers that it would stop at. That's it! I love this redesigned experience of elevators. It solves a numbers of problems. The old style elevators could not predict the demand. Now the system exactly knows how many people are waiting at what floors wanting to go where. This allows the system to optimize the elevator experience based on several variables and criteria such as speed, priority, even distribution, power conservation etc. This also means an opportunity to write interesting algorithms for these elevators.

This is how I want ALL the systems to be - smart, adaptive, and dynamic. Just like this elevator I would like to see the systems, especially the cloud and the analytics, to anticipate the needs of the end users as opposed to following their commands. The context is the key to the success of delivering what users would expect. If the systems are designed to inquire about the context — directly or indirectly, just like asking people to push buttons before they get into an elevator — they would perform more intelligently. Some location-based systems have started to explore this idea, but it's just the beginning. This also has significant impact on designing collaborative recommendation systems that could help the end users find the right signal in the ever increasing noise of social media.

The very idea of the cloud started with the mission to help users with elasticity of the commodity resources without having users to learn a different interface by giving them a unified abstraction. If you had two elevators in a lobby, you wouldn't use this. But, for a high rise with a few elevators, the opportunities are in abundance to optimize the system to use the available resources to provide the best experience to the people, the end users.

Self-configuring and self-healing dynamic systems have been a fantasy, but as the cloud becomes more mature, dynamic capabilities to anticipate the needs of an application and its users are not far fetched. Computing and storage are commodity on the cloud. I see them as resources just like elevators. Instead of people pushing buttons at the eleventh hour I would prefer the cloud take a driver's seat and becomes much smarter at anticipating and managing applications, platforms, and mixed workload. I want the cloud to take this experience to the next level by helping developers develop such adaptive and dynamic applications. I almost see it as a scale issue, at system as well as at human level. If the cloud does promise scale I expect it to go beyond the commodity computing. This is why PaaS excites me more than anything else. That's a real deal to make a difference.