A data migration strategy in six steps

Why do businesses need data migration?

A simple answer to this question is to boost performance and get a competitive advantage. In practice, data migration is usually on the table when businesses introduce new systems: 

• Database migration is needed when a business chooses another database vendor offering more advanced data management features, faster performance, and more opportunities for cost-efficient scaling.
• Storage migration doesn’t always result from a lack of storage space. It’s usually needed when a business adopts more advanced technologies or upgrades their existing system, having to migrate their data to a different storage system, for example, from on-premises to cloud servers.
• App migration is needed when data should be moved from one app or platform to another, within the same vendor ecosystem or between different vendors.

However, data migration becomes a truly pressing matter when there’s a need to overhaul the entire infrastructure by moving from a legacy solution, which has become slow, heavy, and inadequate, to the cloud or between different cloud environments (like in these examples of EHR migration). In most cases, cloud migration embraces database, storage and app migration all at once. 

To understand in what specific cases businesses require data migration, we asked business intelligence consultants from Itransition to share a few examples from their experience.

Email archive migration

A company operating in the email management sector partnered with us to deliver an email archive migration solution and make it compatible with multiple platforms. Within the project, we delivered a tool that supported smooth migration of legacy email archives from/to 25+ archive systems, both on-premises and in the cloud, without compromising data integrity. 

We ensured:

• The system accessibility and business process continuity during the migration.
• Possibilities to analyze the archive content, profile it, perform selective migration, define retention policies, remove duplicates, and filter out obsolete data.
• Data cleanup mechanisms.
• Report generation for data traceability.

As a result, the tool created a competitive advantage for the company by processing large datasets and delivering 10x faster performance than equivalent products.

Pharmaceutical data analytics migration to the cloud 

A US-based multinational company delivering analytics, technology and research services for the pharmaceutical industry approached us to help them keep up with the growing user base of their pharmaceutical data analytics platform and maintain its high performance. The customer’s data assets comprised over 500 million patient records of more than 50 thousand patents. We helped them with a cloud migration strategy and moved their solution from an on-premises server to the cloud.  

As a result, the customer grew the user base and decreased infrastructure costs while ensuring high availability, scalability, and enterprise cybersecurity.

Stage 1. Planning

This stage is by far one of the most crucial ones in your data migration journey. It programs your expectations for the entire project duration, provides a clear route, and lays out the metrics for success. 

Assess the project scope

First of all, you need to perform an exhaustive comparison of source and recipient systems. Together with an IT team and data users who will be affected by data migration, locate the points needing adaptation in the new system and define the minimum amount of data necessary for the new system to start running efficiently. In many cases, it’s rarely necessary to move the entire dataset at once but you can move data in sequences or archive some parts altogether.

Choose a migration approach

Data can be migrated following the big bang approach (doing it in one go in a limited period of time) or the trickle one (migrating in phases while the source and target systems run side by side). When you need to move large datasets, the former approach can seem attractive as it promises to get over with migration in a short period of time. In reality, though, it is not a popular choice for most companies (except those really small ones) as it can cause massive downtime of mission-critical systems, while possible errors can incapacitate business for even longer. 

So most companies migrate data in iterations in spite of the fact that it’s a much more complicated process requiring a well-thought-out design. It also requires more effort from data users, who have to switch between the two systems during the migration, and from data engineers, who have to monitor what data has been migrated and trigger migration updates in case of any data changes. On the bright side, this approach comes with no serious downtimes or operational interruptions if done right and facilitates spotting and solving any errors and issues early in the process. To learn more about choosing the right migration approach, head to our blog post about AWS cloud migration strategies.

Set a realistic project budget and timeline

Timeline and budget estimations will help you understand whether your project is feasible.

The project timeline is calculated based on a few variables. When you understand the amount of data to migrate, you can estimate the time needed to move that data considering the network bandwidth. 

It’s important to understand whether data migration execution will cause any downtime, how to make it as painless as possible (for instance, by planning it for the afterhours or weekends to avoid business continuity interruption), and add this time to your project timeline. 

Consult with your data migration team to estimate the time for data migration tool development or customization, data migration execution, testing, data audit post-execution, and maintenance. Include a safety period for possible contingencies.

To draft your budget, include data migration solution costs, payments to an in-house or outsourcing team throughout the project stages, compliance validation, and possible downtime. Pay attention to the fact that the more data are to be migrated, the higher the project costs. For this reason, it doesn’t make sense to migrate all at once. Instead, review data to understand what can wait and what should be moved in the first place.

Plan the old system retirement

When you don’t plan using the two systems simultaneously after moving and want to fully migrate from your old environment, you should plan its retirement in order to save financial and human resources: 

• Terminate any active software licenses 
• Dispose of unnecessary hardware
• Cancel support and maintenance of the old system
• Plan retraining of your staff

Select a migration tool

If you move petabytes of data of different formats and sensitivity levels, you will need a tool for automating data migration. To get an instrument tailored exactly to your needs, you will likely have to develop it from scratch. Make sure to include this development period to your project timeline. If you plan to use a platform-based solution, check whether it has the necessary features, like easy data mapping with drag-and-drop tools, extensive ETL capabilities, and workflow orchestration.

Build a data migration team

Decide who you’re going to bring to your project. Ideally, you should invite both business users and IT consultants to cooperate. Data users who engage in data-driven decision making and understand the structure and meaning of the source data should share their expertise with the IT team when it comes to data types prioritization for migration, advise what formatting can or can’t be changed, and what workflows are needed and which ones can be scrapped. At the same time, developers can explain the new system specifics and what data is needed to get it going.

If there’s a need for a custom data migration tool, you can cooperate with your in-house IT team if they have the necessary competencies and are available for the stretch of the project, or invite software development consultants to design and build this tool. 

Stage 3. Migration design

Prior to ETL operations, ETL developers and data users should come together for a crucial task—data mapping. It requires preparing detailed mapping rules that will match the fields of the source system to those of the target one. 

Data mapping gets tricky when you need to decide whether migration should be metadata- or content-driven. Metadata usually describes the location of source data, like a file/field/column/table name, and characteristics of each location type, like character, numeric, or date. Content is stored within those fields and columns. 

The metadata-driven approach assumes that content reflects its description, but it’s not always like that. The column described as ‘Email’ can contain a few emails but emails can also be stored in another column under a different name, like ‘Contact Information’. It leads to general assumptions and, by proxy, to mistakes when creating mapping rules and specifications. To avoid mapping errors and multiple iterations, it’s recommended to run content analysis at the stage of data audit and profiling.

Stage 5. Testing and going live

Actually, testing should start from the moment you begin manipulating data and continue throughout all further stages, be it design, execution, or post-migration audit. If you pursue a phased migration approach, you need to test each batch of migrated data to spot any issues early and ensure data quality. Once the errors are fixed, test again. When the migration is complete, verify the migrated data by running unit, system, full-volume and batch-application tests prior to going live. 

When you test the migrated system in its entirety, you can expose a number of unexpected problems never seen during isolated tests. When everything is fixed and tested again and all stakeholders are satisfied with the results, it’s time to finally go live.

Create your data migration strategy now

Considering big data and its business impacts, data will continue to be treated as an asset that companies, even small ones, will accumulate and carry with them when moving to different technological environments. With each move, they will need to make sure they transfer valuable data of high quality only. Here’s where a proven data migration strategy coupled with an experienced team and advanced tools makes companies feel they make a step forward rather than creating more challenges.