Overview

Amazon Web Services is no longer a single option for running workloads and storing data in a cloud. Today there are several competing offerings, such as Exoscale.

Exoscale offers numerous advantages over AWS, including Switzerland-based headquarters and datacenters in a former military complex in Vienna or underground the Alps near Zurich.

Data Migration Challenges

Many cloud-native applications run on extensive amounts of unstructured data, usually stored in a cloud object storage service, such as Amazon S3.

Moving compute workloads is usually a matter of configuring several virtual machines and is not time-sensitive. Moving data, however, poses challenges:

  1. High volumes of data may take days or weeks to migrate, even over fastest internet connections
  2. During data migration users and application still change the data, making already migrate data out-of-sync

While Exoscale gives you better cloud platform, Flexify.IO offers solution to the data migration challenges.

Exoscale S3-compatible API

Simple Storage Service (S3) API, initially developed by Amazon, later become de-facto standard for managing and accessing object data.

Exoscale provides S3-compatible API to Exoscale Object Storage, simplifying migration by eliminating a need to support another API in your application code.

Step-by-step Migration Procedure

Migrating data from Amazon Web Services to Exoscale is made simple with Flexify.IO. To configure and start data migration:

  1. Sign up for an account at https://manage.flexify.io/login-page/signup
  2. Add your AWS S3 storage
    1. On the Welcome page click Configure
    2. Click Add Storage
    3. Choose Amazon S3 form Storage Provider list and enter your AWS Key ID and Secret Key
      You can get your AWS keys at Amazon Web Services Console
    4. Click Verify
    5. Choose bucket with your data from a list of buckets
    6. Click Add Storage to add the storage and close the dialog
  3. Add your Exoscale storage
    1. Repeat the above steps, but this time choose Exoscale from a Storage Provider list
      You can create Exoscale storage bucket and get your Exoscale keys at Exoscale Portal
  4. Flexify.IO will calculate size and number of objects for each storage.
  5. Now you can start data migration (no need to wait for storage stats calculation to finish)
    1. Click Move Data
    2. Choose to migrate from a bucket on Amazon S3 to a bucket on Exoscale
    3. Click Start Migration
    4. Monitor migration progress
Virtual Multi-Cloud Bucket

While the migration is in progress, all objects, both migrated and not yet migrated, are available via a virtual Flexify.IO S3 endpoint.

To access this endpoint – configure your application to use s3.flexify.io instead of s3.amazonaws.com and use keys displayed on Flexify.IO Endpoint Settings card.

Flexify.IO does not store any data - it simply combines multiple cloud storages into one:

  • GET requests are directed to a cloud that actually stores the requested object at the moment - AWS or Exoscale;
  • PUT requests are directed to the storage that you specify as default storage;
  • LIST requests are directed to both clouds and results are merged so that you see a single combined list of objects.

This multi-cloud virtualization mechanism effectively allows you to split your data between multiple clouds and still see it as it were stored in a single cloud.

When migration completes – you can switch the S3 endpoint to Exoscale endpoint (currently sos-ch-dk-2.exo.io).

Advanced Migration Settings

Optionally you may adjust advanced migration options to optimize throughput.

  • Number of slots: the number of parts to split a migration into. Objects are distributed equally between slots. Each slot can be assigned to a separate engine.
  • Maximum engines: the number of machines a migration (it slots) will be assigned to. This defines how many migration slots can be processed simultaneously. Note that we will not assign a migration to more engines that are currently available.
  • Maximum connections per engine: maximum number of objects that each engine will process simultaneously.
  • List objects before migration: defines if Flexify.IO should calculate storage size to display percentage during migration. We recommend that you turn this off if your storage has larger number of objects as it can slow down migration.

Total limit of simultaneous migration streams is the number of engines multiples by the number of streams per engine.

If your data consist of large number of small objects, increasing the number of migration stream usually helps to increase migration speed. However too many streams may lead in storage overload and retries resulting in performance degradation and migration failures.

We are Here to Help

At Flexify.IO we understand that migrating large amount of data between clouds requires careful planning and expertise.

If you plan to migrate over 1 TB of data – feel free to contact us so we can help you to plan the migration, offer special pricing and prepare our infrastructure to optimize your migration throughput.

Comments