Skip to main content

ScanTek

Partner Process
  • Provision new container in blob storage account for migration
  • Provision and distribute credentials to partner so they can SFTP files into blob container
  • Partner uses SFTP client to upload all files to blob container
    • Ideally, the files are structured in the folders the same way they want them uploaded to Revver
      • If they are not though, the CSV files they upload, will need to have details around where to upload
  • Optional: Partner uploads CSV / JSON file to describe metadata fields
    • expected columns: name and type
  • Optional: Partner Uploads CSV / JSON file to describe metadata groups
    • expected properties: name of group, and names of fields
  • Optional: Partner uploads CSV files with structure and metadata to specific location in storage account
    • CSV will have specific columns for certain specific details (node id, desired revver path, blob path, metadata group ids, maybe others?) but all other headers will be metadata field values.
  • Partner uploads JSON file to a specific location to indicate they are done uploading all the data
    • Expected Properties: Import Type (match structure in Blob storage or get structure from CSV), email of Revver user to import them as (we will validate that the user is on the account)
    • Credentials should be revoked at this point probably
    • Migration function app should be kicked off to migrate all the data to Revver
    • Should we generate migration report?
    • When migration is complete blob storage container should be deleted
Work Estimate - 10 days for one engineer (probably want to double for unexpected things and distractions, could be tackled more quickly by a team)

*Note: use utopia API for creating things in Revver

Initial Setup

    • Source Code Repo
    • Build Pipeline
    • Release Pipeline
    • Creating Resources in Azure

Function App (this is a good candidate for a durable function like project hope)

  •  Startup
    • Configuration / Settings
    • where and how will we store mappings and auth tokens
        • can we just use blobs?
        • table storage might be a better fit for this
  • provision new blob container and credentials
  • trigger function for final json to start migration
    • should disable / remove SFTP credentials as well
    • get auth tokens
  • parse metadata fields csv and create in Revver 
  • parse metadata groups csv and create in Revver
  • parse all folder structure create in Revver 
  • Upload all the files
  • Apply Metadata 
  • ? Validate everything was uploaded ?
  • Delete the blob container and delete SFTP credentials (if not doing that at the beginning of the migration)
  • ? Migration Report ?
Additional Questions
  • Could this replace Pangea for Data Imports?
    • Yes, we would just need help them use an SFTP client
  • Could it replace Data Exports for Pangea?
    • No, but we should look at creating a more reliable tool for this
    • I think we could create a tool pretty quickly (especially if we have limited UI) that could do it more effectively and reliably though. All it needs to do is download folders and files.
No Comments
Back to top