Key Insights from the World Forum Event
A panel full of data migration experts gathered virtually on October 20, 2020 for the World Forum Disrupt Data Series webinar to discuss challenges, strategies, and key tactics for migrating databases to the cloud with zero downtime. It was a great discussion, out of which came a number of key insights summarized here. The panel included the following speakers:
- James Rutt (Moderator): Chief Information Officer, The Dana Foundation
- Rahul Subramaniam: Chief Executive Officer, DevFactory
- Robert Leahy: Chief Information Officer, NASA Goddard Space Flight Center
- Robert Williams: Chief Data Officer, Kayne Anderson Capital Advisors
Organizations are rapidly shifting to the cloud to achieve cost efficiencies and power agile business initiatives. For these enterprises, migrating on-premise databases is the first step, but it brings with it several risk factors including potential downtime, data loss, and compromised transactions. According to research published on Data Center Knowledge, database downtime costs companies an average of $7,900 per minute, with an average downtime of 90 minutes.
Insight #1: Prefer parallel migrations for more predictable results
Insight #2: Leverage Database-as-a-Service to reduce complexity
Insight #3: Phased migrations mitigate risk
Insight #4: Security is not just on the perimeter
Insight #5: Decoupling app servers from the database simplifies migrations
Insight #1: Prefer parallel migrations for more predictable results
Robert Williams: “Big bang is sometimes required when you don’t have the luxury of time … but if you can, I’ve always found the parallel method to be much more successful. I’m currently running a completely parallel data warehouse in the cloud (at near real-time).”
Robert Leahy: “It may appear to cost more to run parallel platforms, but in the long run being able to migrate more efficiently with fewer problems affecting customers leads to a better result overall.”
The DevOps approach and tooling make synchronization relatively easy to configure. Given this, the parallel approach provides a realistic approach to early issue identification, continual improvement, and improved expectations management on the new platform.
If you are forced to run a Big Bang migration, have key personnel on hand and ready to assist at a moment’s notice. There will undoubtedly be issues not caught during testing, what the panelists referred to as the “unknown unknowns”, that can impact the customer’s experience.
Insight #2: Leverage Database-as-a-Service to reduce complexity
Rahul Subramaniam: “As a software development organization, focus on your strengths and the areas where you add value to your customers. All of the other pieces are like utilities. For example, you don’t go build your own generators to create electricity.”
Cloud databases such as Aurora have all of the best practices baked into their service. Backups run immediately right out-of-the-box and you can restore anytime you need. Clusters can be configured with the click of a button. Let the experts handle these things so you can focus on adding value to your business. Prefer migrating to managed services rather than simply running your database in the cloud. This will reduce your organization’s complexity unless you truly have unique requirements or special circumstances. One such case occurs when you need to upgrade to a supported database version, but can’t fit that work into the same migration effort.
Insight #3: Phased migrations mitigate risk
Rahul Subramaniam: “The best preparation is to set up migrations such that you are constantly checking. Use blue/green deployments in bursts to incrementally validate, rollback if necessary, and find the remedy.”
Perform validation and gather as many metrics as you can along the way. This enables you to anticipate what might lurk around the next corner. Performance is often an issue at first on the new platform that requires tuning or optimizations, and a phased approach minimizes the blast radius of any potential impacts.
Beyond just the technical challenges, a significant amount of time and energy needs to be devoted to testing, certifications, and assessments. Phased planning can be used to facilitate these test activities in concert with keeping the technical tasks moving forward.
Insight #4: Security is not just on the perimeter
James Rutt: “Of all the security incidents I have had to deal with, the vast majority of them are from the inside.”
Historically there has been a perception that data security equates to locality within the confines of an entity’s physical buildings. This is not the case as external actors and geography are only partial factors. The truth of the matter is that because best practices are already incorporated, cloud providers typically offer a stronger security posture than what is found in many on-premise situations. Additionally, cloud providers are making the use of customer-managed encryption keys much easier to use these days, and this gives organizations an added layer of protection.
Insight #5: Decoupling app servers from the database simplifies migrations
Rahul Subramaniam: “There is typically a strong coupling between the app server and the database server. The biggest challenge is finding the right seams where you can decouple the app servers and database servers. Then the migration becomes simple.”
Tighter coupling between the application and database tiers occurs due to the use of SQL, proprietary database features, and caching strategies with data residing in the application servers. Business logic tends to creep into the database tier, even though the preference is to encapsulate this within the application. Finding the seams and isolating database writes is the key to achieving loose coupling.
What you really need is the equivalent of an NGINX webserver to act as a proxy between the tiers. DevFactory’s strategic investment in the ScaleArc product has been an effective solution for decoupling databases from applications and enabling successful migrations to the cloud.
Looking Ahead
We are currently seeing a very high level of database innovation. Products such as AWS Aurora and Google Cloud Spanner provide orders-of-magnitude increases in performance at fractions of the cost. These technologies are game-changers, and they are affordable on top of that.
The motivation to migrate data to the cloud has never been stronger. A mental model to consider is treating database migrations as you would code refactoring. They are inevitable, so you should plan for them to be as safe, efficient, and cost-effective as possible.
Try ScaleArc or read our whitepaper to find out if ScaleArc is for you.