As a technical consultant for Proact UK, I perform a lot of data migrations to NetApp ONTAP storage. This is often because of a hardware upgrade, storage consolidation, a datacentre move, or older files that just need to be moved to less expensive storage to keep up with soaring data demands.
Although data migration may be a common chore, it doesn't mean it's easy. Migrating data involves a lot more than just ripping out one storage array and plugging in another.
In this blog, I’m going to describe some of the migration approaches I use, as well as the technologies and tools that I employ to ease the pain of data migrations, whether this is moving data from third-party storage to NetApp ONTAP, or from NetApp 7-Mode to NetApp ONTAP.
A data migration plan would typically include the following tasks:
Collect information about hosts, storage, and fabrics in the environment.
Examine the collected data and determine the appropriate migration approach for each host or storage array.
Create and test migration plans, provision destination storage, and configure migration tools.
Migrate the data and perform host remediation.
Validate the new system configurations and provide documentation.
You can read more about migration planning at NetApp.com.
Data Migration Considerations
Most data migrations that cannot be performed non-disruptively tend to be completed out of hours, typically on a weekend/bank holiday where system availability is less critical. This generally gives a big enough window to complete any data moves and ensures that any host remediation and verification tasks can also be completed successfully; more than one migration window may be required.
For planning purposes, certain assumptions can be used to estimate the level of effort and duration of data migrations, but it is advisable to run a number of test migrations in order to gauge an accurate duration to complete your migrations.
Some things to consider when planning your data migrations:
- Storage Vendor you’re migrating from
- Type of data
- Protocols in use
- Size of data
- Number of files
- Migration window and cut-over date
- Network and storage bandwidth
- NetApp OneCollect
NetApp OneCollect is the data collection standard for all NetApp solutions. NetApp OneCollect is a data collection tool that collects data from storage, hosts, and switches. The collected data can not only be used for migration planning, but also for troubleshooting, solution validation, and upgrade assessments.
- NetApp DataCenter Planner (NDCP)
NDCP provides a simple mechanism to extract configuration information from several data collection formats.
NetApp Migration Tools
- 7-Mode Transition Tool (7MTT)
The 7-Mode Transition Tool allows you to transition your data and configurations from 7-Mode to NetApp ONTAP by using either copy-based transition (SnapMirror replication) or copy-free transition (using old disk shelves), and enables you to collect inventory of 7-Mode controllers, hosts, switches, and applications, assess the features and functionalities of these systems, and identify how these features and functionalities work in the NetApp ONTAP version selected for transition. You can then transition your data and configurations from 7-Mode to NetApp ONTAP.
The 7MTT is a great tool and has improved immensely since NetApp ONTAP was released. I’ve used it a ton of times, especially to assess 7-Mode environments to see what does and doesn’t work in NetApp ONTAP. What’s also great about the 7-Mode Transition Tool is that it simplifies the data migration process and enables you to preserve NetApp Snapshot copies, deduplication and compression savings, system configurations, and SnapMirror relationships, making it the best tool available for 7-Mode to NetApp ONTAP migrations. However, it does not allow the simultaneous consolidation of volumes from different 7-Mode sources to an SVM. If you’re migrating from a 7-Mode HA pair, it might not be suitable in some scenarios. That being said, you could always consolidate later with a bit of extra work.
- NetApp XCP
The NetApp XCP Migration Tool is client software that enables fast and reliable any-to-NetApp and NetApp-to-NetApp data migrations.
XCP is ideal for both CIFS and NFS migrations and can handle millions of files. It copies hard links, symlinks, special file types, permissions, ownership, NTFS ACLs, and other attributes, and has good logging, reporting capabilities, and subdirectory granularity, plus three levels of verification.
Note: I have seen instances where XCP doesn't copy symlinks and files with spaces at the end, and although this may not be a big issue, you may have to perform some additional work to ensure all files are copied successfully,
The NetApp XCP Migration Tool is available to download for free from , but it requires a license to work.
Note: Whilst standard licenses are instantly available, if the environment where you are running XCP does not allow outside connections, you’ll need an offline license, which has to go through an approval process.
- NetApp Cloud Sync
Cloud Sync is NetApp’s service for rapid and secure data synchronization, but don’t let the name fool you. As it says on their website, “Whether you need to transfer files between on-premises NFS or CIFS file shares, Amazon S3 object format, Azure Blob, IBM Cloud Object Storage, or NetApp StorageGRID® Webscale appliance, Cloud Sync moves the files where you need them quickly and securely.”
Because Cloud Sync is continuously replicating (based on the schedule you set) and only moves the deltas, transfer times drop from hours to minutes, and even though Cloud Sync runs from the cloud, your data is still secure as it never leaves your network. It’s easy to use, with an intuitive user interface, and has excellent monitoring and reporting capabilities.
You can use Cloud Sync free for 14 days, which might be enough time to perform your data migrations, but after that it’s a pay-as-you-go service.
Cloud Sync is available at cloud.netapp.com.
- NetApp Foreign LUN Import (FLI)
Foreign LUN Import (FLI) is a feature built into ONTAP that allows users to import data from foreign array LUNs to NetApp LUNs in a simple and efficient manner using NetApp’s FlexArray technology to allow your ONTAP controller to act as an initiator in front of a source array and to use that position to copy LUNs block for block from the source to a LUN on the destination ONTAP array. Only a limited number of third-party arrays are supported so be sure to check the IMT for compatibility.
More information can be found in the SAN Migration Using Foreign LUN Import Guide.
- NetApp SVM-DR
It is also worth mentioned SVM-DR, which is a great tool for migrating data between NetApp ONTAP to NetApp ONTAP clusters, as it using NetApp's native SnapMirror replication software, and preserves all settings and minimizes downtime.
Host Operating System or Application-based Migration Tools
- VMware Storage vMotion
- Hyper-V Storage Migration
- Exchange mailbox moves
- Logical volume manager (LVM) based solutions
It some circumstances, it may be a more viable option to use them over a copy based tools.
Third Party Migration Tools
- PeerSync Migration
- Data Dynamics
- Interlock Technology
- Robocopy for Windows
- Rsync for Unix
Data Migration Options
With this in mind, the data migration option(s) you choose has different impacts depending on what you’re migrating.
Examples of the types of data that may have to be migrated are:
- Virtual machine files residing on Hyper-V CSV or SMB volumes, VMware Datastores, and RDMs
- File data (CIFS or NFS)
- Data residing on LUNs or physical server local storage (iSCSI, FC, FCoE)
- Microsoft Exchange databases and log files
- SQL/SharePoint databases and log files
Below are some approaches that I have used to migrate some of the data types mentioned above.
User Files and Application Data
Locking out users can result in a significant and intolerable amount of application downtime. You may also need to create a plan for recreating the file shares, NFS exports, LUN mappings, permissions, compression, encryption, and other settings on the new NetApp storage. This can translate into further hours of downtime even if everything works as planned the first time—and much longer if it doesn’t.
To avoid failure, you should also stop and application or anti-virus processes that lock a file, as these can cause the copy or synchronization process to fail and you may have to restart the copy process from the beginning after finding and stopping the process that’s causing the problem.
Before you proceed with any data migration, it is recommended to install and configure NetApp SnapCenter Server and install the relevant plug-ins on the server. SnapCenter provides centralized control and oversight while allowing you to manage application-specific backup and restore operations for Exchange and SQL.
Note: The contents of your disks and the speed of the SAN and network will make a big difference in how quickly data can be migrated. If the disk is filled with lots of small files, then the migration process will be quite slow. On the other hand, if the disk is only partially filed or filled with large files, it will be lot quicker.
Virtual Machine Files
For VMware, this is Storage vMotion, and for Hyper-V, Storage Migration. Both offer seamless, non-disruptive, live migration of virtual machines files from one storage system to another, with no downtime for the VM or service disruption for end users. This migration occurs while maintaining data integrity.
In-Guest iSCSI Attached LUNS and RDMS
A raw device mapping (RDM) disk is an option in VMware that enables a storage LUN to be directly connected to a virtual machine (VM) from the storage area network. Raw LUNs themselves cannot be moved, as they are raw disks presented from the SAN. However, the pointer files (RDMs) can be relocated if required.
To migrate the data that resides on an In-Guest iSCSI attached LUN or RDM you would create a new disk, and then copy the data from the old drive to the new drive using a copy program within the guest OS. This is time consuming and may be unnecessary. See below.
Various improvements made to vSphere and ESXi have negated the need for In-Guest iSCSI attached LUNS and RDMS. It is now becoming commonplace to convert In-Guest iSCSI attached LUNS and RDMS to virtual disks (VMDK), and therefore may be a more viable option than copying the files to a new disk.
The process to convert an RDM into a VMDK can be achieved using Storage vMotion, however the RDM must be in virtual mode to be non-disruptive.
The process is well documented on vmwarearena.com.
The process to convert an In-Guest iSCSI attached LUN to a VMDK requires the virtual machine to be powered off and the disk reattached as a virtual mode RDM, at which point Storage vMotion can again be leveraged to convert the RDM to a native VMDK.
The process is well documented on Virtual Thoughts Blog.
Migrating SQL and Exchange Resources to NetApp ONTAP Storage Using SnapCenter
Migrating SQL and Exchange databases can be a challenge. but If you're planning to use SnapCenter in your environment, then consider using it to migrate your Exchange and SQL resources to NetApp ONTAP storage.
You can easily migrate databases and logs from one LUN to another LUN using the SnapCenter database migration wizard.
In most cases, SnapCenter may be the most viable option to migrate your Exchange and SQL databases and logs to NetApp ONTAP storage. However, if you're not planning on using SnapCenter in your environment, then you might want to consider using the application GUI, PowerShell, or one of the copy programs mentioned earlier in this document to move them to the new drives instead.
Whichever approach you take, the migration process is disruptive to users as each move would require dismounting the database(s) and taking them offline, so they will not be accessible by your mailbox users during the migration.
If your Exchange environment is in a DAG or your SQL server is in a Failover Cluster, then the preferred data migration approach is to perform the migration non-disruptively by suspending replication on a passive node, using SnapCenter to create the new disks and migrate the files, and reseeding the databases.
For non-DAG or smaller Exchange environments you might also consider using mailbox moves to move your mailboxes non-disruptively.
David Brown is a Technical Consultant for Proact UK and a member of the NetApp A-Team.