Information migration plan. Case research of high-quality, error-proof AWS mission

Why did GOconnectIT search for a technological companion?

GOconnectIT is an organization from the Netherlands creating skilled software program for managing the set up of optical fibers for personal properties and companies.

GOconnectIT is without doubt one of the main corporations within the utilities business, working with 50+ prospects, contractors, and community operators. They focus on offering GEO data, supporting improvements at infrastructural tasks, lowering harm to cables and pipelines, and ensuring that every one needed deadlines and procedures are met.

The corporate had two merchandise for Fiber rollout (FTTH), one among which was designated as end-of-life, intending to maneuver all prospects to the newer product – GO FiberConnect.

The preliminary evaluation confirmed that:

• the previous, legacy system has a really inflexible logic,
• the info is in depth and sophisticated.

The primary drawback we needed to resolve collectively was the configuration which differed relying on GOconnectIT’s prospects and their end-users. Typically a number of prospects work for a similar consumer after which the configurations needs to be equivalent. Sadly in FiberConnect, it diversified and extra knowledge engineering work was needed. 

Preliminary issues and a lesson discovered

Each mission brings a lesson, generally a harsh one. From this expertise, we gained an excellent stronger conviction that ETL tasks require an enormous dedication from each events, area specialists, and a separate and meticulous evaluation section. With no mounted plan, the implementation time will likely be topic to fixed change. As a result of complexity of the mission wants and really sturdy alignment with the consumer, our assumptions have advanced over time which had a direct impression on how we lastly carried out the mission. We grew to become satisfied that it was price spending extra time on the evaluation section. 

At first, it appeared that the duty would consist of remodeling the info from the relational database and getting into it into the suitable goal system database. Information migration wasn’t solely on the relational database degree, but additionally required integration with the brand new system by way of its API. 

For these causes, utilizing present options for database migration working on the SQL degree was out of the query. 

So, how did our knowledge engineering providers resolve this? 

Information migration technique. AWS cloud – we select you!

For the database migration, we picked AWS cloud, and extra particularly the next providers:

• Lambda,
• Step Capabilities,
• SQS,
• Parameter Retailer,
• DynamoDB,
• S3.

We selected this stack as a result of we have already got huge expertise with price limiting on Lambdas. We wanted an answer to find out how briskly the info is processed to the brand new system, and how briskly is knowledge downloaded from the previous system. 

Moreover, we required a service that may permit us to avoid wasting the present state of the migration (errors, which knowledge was migrated and which wasn’t, and so on.), so if needed, we’re in a position to course of solely non-migrated knowledge with out having to delete the previous ones and migrate once more. On this case, we selected DynamoDB as a totally serverless, non-relational database designed to course of large quantities of knowledge (ETL). 

Reusing TCP/IP connections on Lambdas? Why not!

One of many causes some folks could frown upon constructing a posh migration technique that fetches/uploads knowledge from/to a relational database utilizing Lambda features is reusing TCP/IP connections. 

Lambda is a traditional FaaS (Operate as a Service), i.e. a software program improvement mannequin based mostly on code execution on digital machines absolutely managed by a cloud supplier. As FaaS customers, we solely want to supply the operate logic (exported as a handler), which can then be imported by the digital machine and finally run. 

It might look like it’s important to re-establish the TCP/IP connection to the database each time you execute a Lambda operate. As you realize, establishing such a connection is time-consuming and the variety of parallel connections is normally restricted. 

Nevertheless, there are two methods to reuse the identical TCP/IP connections to the database: 

This fashion is just attainable until you employ Amazon RDS or AWS Aurora as your database service. Amazon RDS Proxy permits your serverless functions to share connections established with the database. As soon as the connection is settled, RDS Proxy takes care of it and when one other Lambda needs to hook up with the database, it doesn’t must create a model new connection, however it is going to be served with an already present one. 

• Transferring the TCP/IP connection out of the Lambda handler 

One other method, which is relevant irrespective of which database service you employ (on-premise/cloud/and so on.), is about reusing the identical connection throughout a single Digital Machine the place the Lambda features are operating. 

The key is to take the initialization of the connection outdoors of the exported handler operate. Due to this, the connection will likely be established solely as soon as and can final so long as the digital machine is alive (normally round quarter-hour). 

Extra optimizations like this are described by AWS themselves. In The Software program Home, we use them very intensively, so we extremely suggest the learn.

Information migration course of beneath the microscope 

Lengthy story quick: we constructed an ETL mechanism that retrieves data from the supply database and places it on the queue. Then the ETL processes the queued knowledge and enters it into the goal system. 

Now, onto the main points.

So, what’s with the graph? Let’s analyze it step-by-step as a way to perceive what’s occurring beneath the hood:

1. The method begins with the steps configureMigrateCompletedProjects and configurePlanboardSecondsToWait.

These are some easy ones that configure the parameters that are used later in all the stream. The consumer can both manually present the values when beginning the migration or – if they’re omitted – they are going to be populated with some default values.

2. Step migrateOnlyPlanboard is a traditional AWS Step Capabilities Alternative state.

Name it a easy if assertion. Based mostly on the enter parameter that the consumer specified (or that was populated robotically with a default worth), the stream can cut up into two paths – it may both skip many of the stream and go proper to the MigratePlanboardsFlow, or go gracefully step-by-step. The migrated system is fairly advanced, that’s why knowledge migrations additionally must be configurable as a way to tackle varied use instances. Step MigratePlanboardsFlow is dependent upon the brand new system’s asynchronous jobs completion standing. Generally we all know what’s the standing of those jobs and we will go straight to this step. Generally we don’t – and now we have to traverse the stream’s full path.

3. Steps saveActiveWorkflows, disableActiveWorkflows, and setEmailSendingFalse.  

They’re there to configure the standing of the brand new system earlier than the info switch. They’re just like the warmup earlier than the sport – some minor tweaks and methods earlier than truly shifting knowledge.

4. Steps MigrateFlow, UpdateMigratedAssignmentsFlow, MigrateDocumentsFlow are those the place the true knowledge migration occurs! 

They’re categorized by the entity title that’s at present being migrated. Nevertheless, one among them known as MigrateFlow migrates a number of of the assets which might be logically linked with one another. That’s why they have been positioned collectively beneath a typical step. Additionally, all of them are separate AWS Step Capabilities! Why? Not solely that helps to group frequent components of the method beneath properly named, separate AWS Step Capabilities and monitor the migration course of very exactly from the enterprise perspective, but additionally helps to beat the limitations of historical past occasions per step features. 

Throughout AWS re:Invent 2022, a distributed map was added as a stream which is great for orchestrating large-scale parallel workloads. Each iteration of the map state is a baby execution with its personal occasion historical past – which implies you received’t run right into a 25,000 historical past occasions restrict per father or mother’s Step Operate execution.

 

5. The following step – waitForProcessingPlanboard – is an AWS Step Capabilities Wait state. 

Bear in mind after we instructed you about these asynchronous jobs within the new system that must be executed earlier than going ahead with the migration? If we all know that these jobs haven’t been finalized but, now we have to attend a bit. Right here goes the… wait step. Badum-tss!

6. MigratePlanboardFlow is one other stage the place knowledge migration occurs. 

That is this nasty step that’s depending on these asynchronous jobs execution. Since we all know that at this migration stage, all of them needs to be already executed, we will proceed ahead.

7. Final however not least, GenerateReportFlow step is triggered. 

It generates a set of stories which include a abstract of the migration course of:

• how a lot knowledge was efficiently transferred from the previous to the brand new system, 
• how a lot knowledge was not migrated, 
• why the remaining knowledge couldn’t be migrated
• the length of the migration course of, and so on.

Working operations per sub-project

Since we needed to migrate not one, however many databases, we couldn’t merely run one Lambda Operate and command it to carry out the migration course of for all the databases. That’s due to some exhausting Lambda-level limitations like most execution time. 

Due to that, we had to consider some normal resolution that might undergo all the databases and run the migration course of for them. The diagram beneath illustrates our method (which is – you guessed it – one other AWS Step Operate).

ProcessFlow is run utilizing a dynamic variable holding the ARN from the Step Operate that will likely be known as. This not solely permits us to group the method logically – by sub-projects – but additionally to scale back the variety of historic occasions in a single Step Operate (the restrict talked about earlier within the article).