Partner Migrations
ScanTek
Partner Process
- Provision new container in blob storage account for migration
- Provision and distribute credentials to partner so they can SFTP files into blob container
- Partner uses SFTP client to upload all files to blob container
- Ideally, the files are structured in the folders the same way they want them uploaded to Revver
- If they are not though, the CSV files they upload, will need to have details around where to upload
- Ideally, the files are structured in the folders the same way they want them uploaded to Revver
- Optional: Partner uploads CSV / JSON file to describe metadata fields
- expected columns: name and type
- Optional: Partner Uploads CSV / JSON file to describe metadata groups
- expected properties: name of group, and names of fields
- Optional: Partner uploads CSV files with structure and metadata to specific location in storage account
- CSV will have specific columns for certain specific details (node id, desired revver path, blob path, metadata group ids, maybe others?) but all other headers will be metadata field values.
- Partner uploads JSON file to a specific location to indicate they are done uploading all the data
- Expected Properties: Import Type (match structure in Blob storage or get structure from CSV), email of Revver user to import them as (we will validate that the user is on the account)
- Credentials should be revoked at this point probably
- Migration function app should be kicked off to migrate all the data to Revver
- Should we generate migration report?
- When migration is complete blob storage container should be deleted
Work Estimate - 10 days for one engineer (probably want to double for unexpected things and distractions, could be tackled more quickly by a team)
*Note: use utopia API for creating things in Revver
Initial Setup
-
- Source Code Repo
- Build Pipeline
- Release Pipeline
- Creating Resources in Azure
Function App (this is a good candidate for a durable function like project hope)
- Startup
- Configuration / Settings
- where and how will we store mappings and auth tokens
-
- can we just use blobs?
- table storage might be a better fit for this
-
- provision new blob container and credentials
- trigger function for final json to start migration
- should disable / remove SFTP credentials as well
- get auth tokens
- parse metadata fields csv and create in Revver
- parse metadata groups csv and create in Revver
- parse all folder structure create in Revver
- Upload all the files
- Apply Metadata
- ? Validate everything was uploaded ?
- Delete the blob container and delete SFTP credentials (if not doing that at the beginning of the migration)
- ? Migration Report ?
Additional Questions
- Could this replace Pangea for Data Imports?
- Yes, we would just need help them use an SFTP client
- Could it replace Data Exports for Pangea?
- No, but we should look at creating a more reliable tool for this
- I think we could create a tool pretty quickly (especially if we have limited UI) that could do it more effectively and reliably though. All it needs to do is download folders and files.