DSS Noteworthy information about container data migration
Usually every 5 to 6 years we need to refresh the storage hardware that drives the LRZ Data Science Storage. Usually this also means that we need to move the DSS containers on the old pool to a new pool. While we work hard to make this transition as smooth as possible it is not completely transparent to data curators and users though. In the following we describe the current migration process step by step and outline the required data curator and user responses in each of theses steps.
The DSS Data Migration Procedure
Let's say we have a DSS Container called pr74qo-dss-0000 on the DSS pool LRZ_PILOT (which path is /dss/dssfs01) and this container needs to be migrated to the DSS pool SUPERMUC_NG (which path is /dss/dssfs02).
| Step | What will happen | Required Data Curator Actions | Required User Actions |
|---|---|---|---|
| 1 | In the first step we will create a Read Only twin of the container at /dss/dssfs01/pr74qo/pr74qo-dss-0000 on the new pool under /dss/dssfs02/pr74qo/pr74qo-dss-0000 and will once pull over all data stored at /dss/dssfs01/pr74qo/pr74qo-dss-0000 to /dss/dssfs02/pr74qo/pr74qo-dss-0000 | If you use Globus Sharing on this container, please note down all Globus Sharing ACLs you defined as these will be lost during the migration. | None |
| 2 | In the second step we will usually announce a system maintenance with downtime as it is required that all IO on the container source path So after all IO has stopped on the container source path we will migrate all NFS exports and the Globus Shared Endpoint pointing from the container source path to the container target path Then we will switch the container target path from Read-Only to Read-Write whereby each file in the container still remains a twin of the file on the source container until the file has been written to the first time on the target container. After that, we will migrate any backups from the container source path to the container target path. And finally we will trigger final sync from the container source to the container target that will pull over all data that has been changed since the first sync, carried out in step 1. Last but not least we will remove the "twin relationship" between the source and the target container path completely and the migration maintenance is finished. | None | Stop all IO on the container |
| 3 | After the migration maintenance, data curators and users need to perform some adjustments. | If you used Globus Sharing on this container, please recreate the Globus Sharing ACLs you saved in step1. | Make sure your workloads are now pointing to the container target path instead of the container source path If you use NFS mounts of the container please adjust your NFS mounts to point to the new NFS export server and container target path |