Simple Workflow Service - Amazon Adding One Enterprise Brick At Time

Yesterday, Amazon announced a new orchestration service called Simple Workflow Service. I would encourage you to read the announcement on Werner's blog where he explains the need, rationale, and architecture. The people I spoke to had mixed reactions. One set of people described this as a great idea and were excited that the developers can now focus on writing domain-specific code as opposed to writing plumbing code to orchestrate their actual code. The other set of people felt that this service creates a new cloud lock-in making it difficult for the developers to switch from one cloud to another as well as being able to interoperate at the orchestration level.

I believe this is a brilliant idea for a variety of reasons. Orchestration has always been painful. Ask the developers who have been involved in managing task execution across a cluster that required them to code for load balancing, handling exceptions, restarting hung processes, tracking progress etc. This is not a core competency the most developers have but they do end up writing such code due to lack of better alternative. The frameworks such as WS-BPEL were not designed to run in cloud-like environments and there has been no single standard REST orchestration framework out there that people could use.

From a vendor's perspective, I admire Amazon's ability to keep innovating via such services that differentiate them as a leading cloud vendor. As computing becomes more and more commodity, competing based on price alone isn't a good idea. If you're a cloud vendor you need to go above and beyond the traditional IaaS attributes even though you excel in all of them. I also see PaaS eventually bleeding into IaaS as IaaS continues to become a commodity. As far as PaaS goes, federated or otherwise, we're barely scratching the surface.

I don't see this service as a cloud lock-in but it certainly makes EC2 more attractive and sticky. I would be concerned if Amazon were to force the developers to use their SWS for orchestration. This is their version of how they think orchestration should be done and the developers can opt in if they want. And kudos to them to think beyond their cloud. The folks who worry about cloud lock-ins also talk about Amazon not following the standards. I believe that we should not create standards for the sake of creating standards. I am a believer in first showing that something works and later, if there's enough interest, figure out a way to standardize it. All these talks about standard-first even before you write that first line of code doesn't make any sense.

It's yet to be seen how this service turns out, but this is a huge step forward for getting more enterprise software customers onboard. Orchestration is one of the most chronic problems of enterprise software and with the challenges of a hybrid landscape to be able to orchestrate across on-premise and cloud-based solutions, this service is certainly a step in the right direction. Right Scale has been using a Ruby workflow Ruote for their workflow needs and now they orchestrate these workflows using SWS  to achieve fault tolerance and concurrency. As you can see, Amazon has opened up a gold mine for start-ups. The back-end execution has always been challenging. Now, there is an opportunity to write your own enterprise grade workflow engine or scheduler that runs in the cloud.

Wrong Side Of The IT Ecosystem

I find it ridiculous that people are blaming Apple for job creation in China as opposed to in the US. People are also debating how US might in-source some of these manufacturing jobs to compete with China who has sophisticated manufacturing abilities and large skilled labor force supporting these operations. They are all missing the point. This is a wrong debate.

The US lost manufacturing jobs to other countries a long time ago. I find it amusing that people expect the high-tech companies such as Apple to create manufacturing jobs in the US. If Apple were to even consider this option we would not have seen the tremendous success of Apple as a company and its products. What Apple created is an ecosystem of people and companies that are doing amazing things with their platform and their devices. It's a different kind of ecosystem and America should focus on that innovation as opposed to bringing those manufacturing jobs back.

On one side we are whining about the loss of manufacturing jobs and on the other side we have shortage of skilled IT workforce. Try hiring a good developer in the Silicon Valley and you'll understand what I mean. And yet as a nation we are behind in retraining our existing workforce, attracting students to engineering majors, and fixing our immigration policy for highly skilled foreign workers to meet the increased demand of IT-driven jobs. And, of course, while we wait, Apple is quadrupling its IT investment in India.

America should not play the manufacturing game with China or for that matter with anyone else. We are at such a loss. Let's play the game that we know we can win — technology-driven innovation. When I work with customers' on daily basis I come across so many opportunities that we are not looking at. We can use the technology, that we have built, to our advantage in the industries such as healthcare, agriculture, public sector etc. A combination of cloud and mobility could take us long way.

We're looking at the wrong side of the IT ecosystem. I don't expect the hi-tech companies to hire low-tech workers in the US. But I do expect hi-tech companies to create jobs in the US at the other end of the ecosystem via the opportunities to consume their technology and innovate in a different sector. A lot of people are missing this point. I'm talking about an ecosystem where Apple has paid out more than $4 billion to the developers. Why are we not talking about these jobs? Apple has more than $100 billion in cash, but what doesn't get much discussed is that a large part of this cash is overseas. Given the current US tax laws, Apple can't/won't bring this cash back into the US. This might make Apple acquiring or investing overseas. We do have an opportunity to reform the tax laws to deal with such a global situation (that we never encountered before) to encourage the hi-tech companies to invest into R&D in the US and not overseas.

When you look at the big picture, having a job is merely one piece of contributing to good standards of living. What about access to affordable healthcare and college education? There's a significant opportunity to apply technology built in America to innovate in these areas. We are barely scratching the surface of what's possible in healthcare as well as in education. We are living in such an antiquated and inefficient system.

Another industry that has seen less than desired technology innovation is agriculture. Take a trip down to central California to see the potential. At 2008 Olympics in China, Michael Phelps winning 8 gold medals was obviously the biggest highlight for us, but people in Watsonville were super excited because China allowed the US to export Watsonville strawberries for the Olymipcs. Recently, India relaxed the laws (that are still being challenged) to allow 100% foreign investment in the retail sector opening up the doors for Wallmarts of the world. Any guess what's the biggest challenge in retail operations in India? A non-existent cold supply chain and lack of reliable infrastructure. We take a lot of things for granted — nationwide freeways, strong governing bodies such as FDA, and size of the country. We do have an opportunity to excel in certain agriculture areas and employ a lot of Americans. We need to recognize what our real strength is and look forward as opposed to look backwards.

I am a geek and a technology enthusiast, and perhaps a little naive. But, I know for sure, we aren't pushing the technology envelope as much as we should.

Photo: Looking Towards Tomorrow