Last year, I made some predictions about how the cloud and data would change the technology landscape. This included the now clear rise of SQL, the consolidation of cloud application vendors, the end of the war between the public and private cloud, and the growth of analytics fueled by machine data.
So what might be in store this year? From data walls to Spark divorcing Hadoop, this year looks to be another groundbreaking year in technology.
Do or die for big old tech
The day of reckoning is coming for enterprise tech companies. Everybody knows that the cloud has completely changed the game, but the big old tech companies - Oracle, IBM, HP, SAP, Symantec, Cisco, NetApp, Dell/EMC, Teradata, et al. - are still stuck in the old world. Dell and EMC have accepted their destiny and have retreated into privacy in order to milk the legacy cows. HP has split up and is taking the same approach, albeit in a public way, at least for now. This is actually a reasonable financial strategy given that legacy cows have a long lifetime. But it's not very exciting.
Facing stagnant and declining revenue, others, such as Oracle and IBM, are trying to have it both ways, milking legacy businesses while dabbling with cloud offerings that are often barely more than a managed services veneer. Companies like NetApp and Symantec are lost in the fog. SAP is buying their way to the cloud, which may be pretty sensible. Ironically, the one big old tech company that appears to have made the change is Microsoft. It's easy to give Satya credit for this, and he is doing a great job. But I honestly have to say that Steve and Bill started the cloud transition at Microsoft over five years ago. It would be great to see other tech companies make this same shift but their time is running out.
Public cloud wins. But who loses?
The year of the private cloud never happened. What we see instead are better automated, virtualized data centers. The technical and political complexity of multi-tenant shared storage and networking is the bane of private cloud deployments. But maybe this is ok as the public cloud has emerged as a better alternative anyway.
In terms of winners, of course there is Amazon. But Microsoft has the enterprise chops to compete and will invest whatever it takes to be in the game. Google is the wildcard, as they possess the technical ability to succeed but completely lack any understanding of the enterprise. Perhaps the recent appointment of Diane Greene as cloud czar will change that. You will know that Google is serious if the enterprise cloud is broken out as a separate business unit under Alphabet and they announce that they are hiring 3,000 enterprise sales people. Without those changes, Google will continue to flounder.
So who loses? Everybody else.
What do Donald Trump and EU bureaucrats have in common?
Answer: They both want to build a wall
The difference is that The Donald wants to keep the people out, and the EU wants to keep the data in. While no physical walls will be built, the insistence by European governments to keep data within their borders will be reinforced in 2016.
The European court ruling that invalidated Safe Harbor is wreaking havoc on both sides of the Atlantic. There is motivation to create a political solution but no agreement on what that solution should be. Given the difference in viewpoints across different European nations and in the United States, a political solution is unlikely.
Expect continued turmoil. Some are seeking shelter using legal "model clauses," which may provide a short-term solution. But longer-term, don't expect these to stand. For now, the only safe approach is to keep European personal data in Europe.
Adding to the insanity, leaving European data in Europe even may not be enough. In a current lawsuit, the U.S. government claims that Microsoft needs to provide U.S. officials with emails related to a narcotics case even though these emails are stored in an Irish data center.
If Microsoft loses this case, the implications are profound. It would allow the U.S. government to force any American company to turn over data regardless of where it is stored, thus destroying global trust in U.S. technology. Paradoxically, this crazy case would, by implication, allow other governments to demand access to data that is stored in the U.S. Does our government really want to set a precedent that would allow China to demand access to data stored in the U.S. just because the data center was run by a Chinese company?
One final thought - when Microsoft recently announced the opening of the Azure datacenter in Germany, they did this in cooperation with Deutsche Telekom. DT acts as a trustee, providing a shroud of protection that prevents Microsoft from complying with future U.S. court orders for data. Very clever, Microsoft. If the Irish case goes south, you can expect to see Amazon and others follow Azure's lead on this.
Spark divorces Hadoop
Hadoop is a complex beast. The software equivalent of LEGOs, Hadoop is a collection of open source projects. Considerable assembly is required and multiple outcomes are possible. Like LEGOs, the end result might look cool but it is probably less functional than a purpose-built solution.
Spark is different. It provides an efficient, general-purpose framework for parallel execution. This is very useful in today's world where data analysis often requires the resources of a fleet of machines working together. While Spark is still relatively immature, it has the potential to evolve into the standard framework and API for parallel algorithmic analytics and machine learning.
Today, Spark is part of Hadoop distributions and is widely associated with Hadoop. Expect to see that change in 2016 as Spark goes its own way, establishing a separate, vibrant ecosystem. In fact, you can expect to see the major cloud vendors release their own Spark PaaS offerings. Will we see an Elastic Spark? Good chance.
As always, what actually happens this year will be even more exciting.