Tuesday, August 22, 2017

Achieve Enterprise Integration with AWS Lambdas, SWF and SQS

In recent days, We were asked to do an ETL flow using Amazon Web Services. Because we excel in Enterprise Integration we had a particular design in mind to make it happen. The job was pretty simple:
  1. The trigger was a file placed in a particular S3 bucket.
  2. Take the S3 object metadata of the file as the input of the job.
  3. Read the file and package the records in pages, each page is sent asynchronously as a message. This technique is to increase parallelism in the job processing since the files contain one million records in average.
  4. Consume all pages asynchronously and upload them as micro-batches of records into a third-party system via a Restful API.
  5. Other tasks to complete the use case like recording the completion of the job in a database.
On top of these basic requirements we had to make sure the system was robust, resilient and as fast as possible while keeping low the costs of the different systems.

We chose to use different services from Amazon Web Services for this: S3, Simple Workflow (SWF), Simple Queue Service (SQS) and Lambda.

Here a diagram of the solution (click on the image to see it bigger).

Solution diagram
Solution diagram


Why Simple Workflow (SWF)?

As you can see in the diagram, every task is executed by a Lambda function, so why involve Simple Workflow? The answer is simple: We wanted to create an environment where the sequence of task executions was orchestrated by a single entity, and also be able to share with the different tasks the context of the execution.

If you think of this, we wanted to have something similar to a Flow in a Mule app (MuleSoft Anypoint platform).

It is important to highlight that AWS has some specific limits to execute Lambdas like one Lambda function can only run for a maximum of 5 minutes. Due to these limits, we had to break the tasks into small but cohesive units of work while having a master orchestrator that could run longer than that. Here's where the shared context comes useful.

Note: There's another service that plays very well on the serverless paradigm as opposed to SWF, Step Functions, but at the time We were working on this task it was still in Beta, hence not suitable for production. There is a follow-up post about full Serverless integration that will include Step Functions.

Challenges and recommendations


While working with SWF and Lambdas, We learned some things that helped us a lot to complete this assignment. Here I'll show you the situation and solution that worked for me.

Invoke Lambdas from activities, not workflow workers


One thing you should know about working with SWF is that every output of an activity returns as a Promise to the workflow worker - very similar to a Promise in JavaScript. This Promise returns the output as a serialized object that you need to deserialize if you want to use it as an input for a Lambda function execution directly from the workflow worker. This overhead can be very cumbersome if you use it frequently. In your lambdas you're supposed to work with objects directly, not serialized forms.

Here my first advice, even though you can invoke a Lambda function from within a workflow worker don't do it, instead use an Activity worker. This way each workflow worker implements a unit of work that calls an Activity worker which in turn calls a Lambda function internally. Why? Because in the Activity worker you will be able to use a proper object to pass to the Lambda as an input parameter. This technique requires you to deal with some extra plumbing in your SWF code since you'll need one Activity per Lambda, but in the end, this provides you a very flexible and robust mechanism to exchange information between SWF and Lambdas.

See this sequence diagram to understand it.

Workflow, activity and lambda sequence diagram.
Workflow, activity and lambda sequence diagram.


Wrap your payload in a Message object


All in all, we are talking about Enterprise Integration and one of the central pieces is the message. In order to uniformly share information between the workflow and the different Lambdas, it's better to standardize this practice by using a custom Message object. This Message must contain the workflow context you want to share and the payload. When the Lambda functions are called, they receive this Message object that they use to extract the information required to perform the task fully with no external dependency.

Decompose large loads of data into small pieces


As mentioned before, Lambdas are supposed to run small tasks quickly and independently, therefore they have limits that you should be aware of, such as execution time, memory allocation, ephemeral disk capacity, and the number of threads among others. These are serious constraints when working with big amounts of data and long running processes.

In order to overcome these problems, we recommend decomposing the entire file content into small pieces to increase task parallelism and improve performance in a safe manner - actually, this was one of the main reason to use Lambdas since they auto-scale nicely as the parallel processing increases. For this, we divided the file content into packages of records as pages, where each page can contain hundreds or thousands of records. Each page was placed as a message in an SQS queue. The size of the page must consider the limit of 256 KB per message in SQS.

Keep long running processes in Activities, not Lambdas


As you see in the diagram above, there's a poller that is constantly looking for new messages in the SQS queue. This can be a long running process if you expect dozens of thousands of pages. For cases like this, having activities in your flow is very convenient as you can have an activity running for up to one year, this contrasts highly with the 5-minute execution limit of a Lambda function.

Beware of concurrency limits


Consider the scenario where you have an Activity whose purpose is to read the queue and delegate the upload of the micro-batches to an external system. Commonly, to speed up the execution you make use of threads - note I'm talking about Java but other languages have similar concepts. In this Activity, you may use a loop to create a thread per micro-batch to upload.

Lambda has the limit of 1024 concurrent threads, so if you plan to create a lot of threads to speed up your execution, like uploading micro-batches to the external system mentioned above, first and most importantly, use a thread pool to control the number of threads. We recommend do not create instances of Thread or Runnable, instead, create Java lambda functions for each asynchronous task you want to execute. Make sure you use the AWSLambdaAsyncClientBuilder interface to invoke Lambdas, the ones in AWS, asynchronously.


Conclusion


This approach was particularly successful for a situation where we were not allowed to use an integration platform like Mule. It is also a very nice solution if you just need to integrate AWS services and move lots of data among them.

AWS Simple Workflow and Lambda work pretty well together although they have different goals. Keep in mind that an SWF application needs to be deployed on a machine, like a standalone program, either in your own data center or maybe an EC2 instance, or another IaaS. 

This combo will help you to orchestrate and share different contexts, either automated through Activities or manual by using signals, but if what you need is isolated execution and chaining is not relevant to you, then you could use Lambdas only, but the chained execution will no truly isolate them from each other and the starting Lambda may timeout before the Lambdas functions triggered later in the chain finish their execution.

Moreover, every time you work with resources with similar limitations like AWS Lambdas, always bear in mind the restrictions they come with and design your solution based on these constraints, hopefully, in Microflows.  Have a read on the Microflows post by Javier Navarro-Machuca, Chief Architect at IO Connect Services.

To increase parallelism we highly recommend using information exchange systems such as queues, transient databases or files. In AWS you can make use of S3, SQS, RDS or DynamoDB (although our preference is SQS for this task)

Stay tuned as we're a working on a solution that uses Step Functions with Lambdas rather than Simple Workflow for a full Serverless solution integration.

Happy reading!

Resources


Enterprise Integration Patterns - http://www.enterpriseintegrationpatterns.com/
Amazon Simple Workflow - https://aws.amazon.com/swf/
Amazon Lambda - https://aws.amazon.com/lambda/ 
Amazon Simple Queue Service - https://aws.amazon.com/sqs/


80 comments:

  1. Hi there, awesome site. I thought the topics you posted on were very interesting. I tried to add your RSS to my feed reader and it a few. take a look at it, hopefully I can add you and follow.



    Ionic Framework Development Company India

    ReplyDelete
  2. Nice post,it's very informative.i found the best information.I updated my knowledge with this blog.it can help me to crack GIS jobs in Hyderabad.

    ReplyDelete
  3. Those guidelines additionally worked to become a good way to
    recognize that other people online have the identical fervor like mine
    to grasp great deal more around this condition.



    AWS Training in Bangalore


    AWS Training in Bangalore

    ReplyDelete
  4. Appreciation for really being thoughtful and also for deciding on certain marvelous guides most people really want to be aware of.

    AWS Training in Bangalore|

    ReplyDelete
  5. I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.
    amazon-web-services-training-institute-in-chennai

    Best Hadoop Training Institute In chennai

    ReplyDelete
  6. informative blog thanks for providing such a great information.
    Aws Training in Hyderabad

    ReplyDelete
  7. Very Good Information. Thanks for this great share.I like your blog, I sincerely hope that your blog is a fast-growing traffic density, and to help promote your blog, we hope you blog updates and place can always be colorful.
    AWS Architecture

    ReplyDelete
  8. Nice Post I learned a lot From the Post Thanks for sharing, learn the most ON-DEMAND software Training in Best Training Institutions
    Instructor-LED Salesforce Online Training

    Best Salesforce Online Training
    Professional Salesforce CRM Training

    ReplyDelete
  9. Nice post keep do posting The Info was too good, for more information regarding the technology Click

    aws training in chennai

    selenium training in chennai

    ReplyDelete
  10. I am writing a couple of phrase that almost all of us use on a regular basis. And I am writing to let you know all of us under-use or misuse it.This is great blog. If you want to know more about this visit here AWS Cloud Certification.

    ReplyDelete
  11. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts, have a nice weekend!
    Online training in USA

    ReplyDelete
  12. Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work

    DevOps is currently a popular model currently organizations all over the world moving towards to it. Your post gave a clear idea about knowing the DevOps model and its importance.

    Good to learn about DevOps at this time.


    devops training in chennai | devops training in chennai with placement | devops training in chennai omr | devops training in velachery | devops training in chennai tambaram | devops institutes in chennai | devops certification in chennai | trending technologies list 2018

    ReplyDelete
  13. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
    SAP Hybris Online Training

    SAP Fiori & ui5 Online Training

    ReplyDelete
  14. Amazon Web Services (AWS) is the most popular and most widely used Infrastructure as a Service (IaaS) cloud in the world.AWS has four core feature buckets—Compute, Storage & Content Delivery, Databases, and Networking. At a high level, you can control all of these with extensive administrative controls accessible via a secure Web client.For more information visit.
    aws online training
    aws training in hyderabad
    aws online training in hyderabad

    ReplyDelete
  15. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    Python Training in Chennai

    ReplyDelete
  16. Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating mulesoft online course hyderabad

    ReplyDelete
  17. Thinking how to make money? Come to us and win now good slot online Play and win always and with us.

    ReplyDelete
  18. Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

    machine learning training center in chennai
    machine learning with python course in chennai
    machine learning course in chennai

    ReplyDelete
  19. This comment has been removed by the author.

    ReplyDelete
  20. Thank you for taking the time and sharing this information with us. It was indeed very helpful and insightful while being straight forward and to the point.
    Java Training in Chennai
    Java course in Chennai
    Angularjs Training in Chennai
    Selenium Training in Chennai
    German Classes in Chennai
    Java Training in Anna Nagar
    Java Training in Velachery

    ReplyDelete
  21. Thank you for sharing your article. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the same to other forums.

    best openstack training in chennai | openstack course fees in chennai | openstack certification in chennai | openstack training in chennai velachery

    ReplyDelete
  22. Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative.
    Workday HCM Online Training!
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  23. Гибкая світлодіодна лента лучшая на рынке, я обычно покупа в Экодио, лучше не найти.

    ReplyDelete
  24. Amazing! I like to share it with all my friends and hope they will like this information.
    Regards,
    Python Training in Chennai | Python Programming Classes | Python Classes in Chennai

    ReplyDelete
  25. marvellous!i really want to say that everyone will get new thoughts after read your post and you have narrated beautiful ideas
    Python Training in Chennai
    IOS Training in Chennai
    Android Training in Chennai
    Android Training in T Nagar

    ReplyDelete
  26. QuickBooks Payroll Support Phone Number management quite definitely easier for accounting professionals. There are so many individuals who are giving positive feedback if they process payroll either QB desktop and online options

    ReplyDelete
  27. QuickBooks Premier is very simple to utilize but errors may usually pop up during the time of installation, at the time of taking backup, while upgrading your software to your latest version etc. The support team at QuickBook Customer Support Number is trained by well experienced experts that are making our customer care executives quite robust and resilient.

    ReplyDelete
  28. The QuickBooks Payroll Tech Support Number team at site name is held responsible for removing the errors that pop up in this desirable software. We care for not letting any issue can be purchased in in the middle of your work and trouble you in undergoing your tasks.

    ReplyDelete
  29. Most of us is responsible and makes sure to deliver hundred percent assistance by working 24*7 to meet your requirements. Go right ahead and mail us at our QuickBooks Customer Support Number email id if you are in need. You could reach us via call at our toll-free number.

    ReplyDelete
  30. Our customer care executives have a great deal of experience and are sharp as well as smart in figuring out the actual cause and optimal solution of any error that you may face. Call QuickBooks Enterprise Support number anytime for our uninterrupted support as we are available 24*7for your help.

    ReplyDelete
  31. Our support services are not limited to your above list. A large number of folks are successfully availing the outstanding options that come with our technical support desk for QuickBooks. With a massive client base, we now have achieved the 100% satisfaction rate from customers around the globe. You too can avail our support services on just a call. Don’t hesitate to contact us at QuickBooks Tech Support Number in case you encounter any kind of technical bug in QuickBooks.

    ReplyDelete
  32. The error will likely not fix completely unless you comprehend the main cause connected with problem. As company file plays a really crucial role in account management, so that QuickBooks Tech Support Phone Number becomes only a little tough to identify.

    ReplyDelete
  33. For such kind of information, be always in contact with us through our blogs. To locate the reliable way to obtain assist to create customer checklist in QB desktop, QuickBooks online and intuit online payroll? Our Quickbooks Support service may help you better.

    ReplyDelete
  34. But, being a regular business person, working on professional accounting software, like QuickBooks Support Phone Number, is certainly not always easy. Thus, users may have to face a wide range of issues and error messages while using the software; whenever you feel something went wrong together with your accounting software and should not find a way out, you could get technical support from our experts’ team, day time and night to fix any issues linked to QuickBooks.

    ReplyDelete
  35. This saves huge cost. All experts can take place. QuickBooks Support Phone Number team operates 24/7. You receive stress free. Traders become free. No one will blame you. The outsourced team will see all.

    ReplyDelete
  36. Fix all of the issues for QuickBooks Payroll Tech Support Number Desktop Payroll Basic, Standard, & Payroll Assisted. Look to the above number to get hold of our ProAdvisor to possess support for QuickBooks online Payroll support, Intuit Online & Full Service Payroll.

    ReplyDelete
  37. We plan to give you the immediate support by our well- masterly technicians. A group of QuickBooks Support dedicated professionals is invariably accessible to suit your needs so as to arranged all of your problems in an attempt that you’ll be able to do your projects while not hampering the productivity.

    ReplyDelete
  38. For those who have recently come across a peculiar QuickBooks error, specifically, QuickBooks Error 3371, Status Code 11118 “QuickBooks could not load license data. This may be due to a missing or damaged file.” with a status code of 11118, it's not just you. This problem is prevalent amidst lots of users who use QuickBooks for accounting purposes.

    ReplyDelete
  39. QuickBooks Payroll Support Phone Number will help you in calculating the complete soon add up to be paid to a worker depending until the number of hours he/she has contributed of their work. The amazing feature for this application is direct deposit and this can be done at any time one day.

    ReplyDelete
  40. QuickBooks Enterprise Support Phone Number has almost eliminated the typical accounting process. Along with a wide range of tools and automations, it provides a wide range of industry verticals with specialized reporting formats and tools.

    ReplyDelete
  41. Intuit Official support number you will probably confused when a so many number comes up in the search results ,because Intuit is dealing with so many products that why each product and each region they having the different Tech Support official . But if you are in hurry and business goes down due to the QB error you can ask for Quickbooks Consultants or Quickbooks Proadvisors . If you want to consult with the QuickBooks experts than AccountsPro Quickbooks Support is for you !

    ReplyDelete
  42. But don’t worry we have been always here to aid you. As you are able to dial our QuickBooks Payroll Tech Support Number. Our QB online payroll support team provide proper guidance to fix all issue connected with it. I will be glad that will help you.

    ReplyDelete
  43. The quickbooks twenty four hours customer support team is available 247 at toll-free and makes sure that its customers are provided with aid on time. Since the customers are of numerous backgrounds and differing types of businesses, their comprehension of accounting may or may not be as much as the mark. Although QuickBooks has left no stone unturned to utilize this software without having any prior familiarity with accounting, often times you can find issues that can’t be solved in single handed manner. The quickbooks online phone support at toll-free number is the best support team that you should go for while using QuickBooks. Their QuickBooks Support Phone Number USA provides top notch assistance at any point of time.

    ReplyDelete
  44. On September 22, 2014, Intuit publicized the release of QuickBooks 2015 with types that users have already been completely demanding through the past versions. Amended income tracker, pinned notes, better registration process and understandings on homepage are the large choice of general alterations for several versions of QuickBooks 2015. It can benefit for QuickBooks Enterprise Support Phone Number to have technical help & support for QuickBooks.

    ReplyDelete
  45. Any user can look for now available payroll update when you go to “employee” menu, selecting “get payroll updates” after which it option “update”. In the window “get payroll updates” you can examine whether you're making use of the latest updates or simply not. For almost any information or update, you can contact Quickbooks Support Phone Number.

    ReplyDelete
  46. Our technical help desk at not simply supplies the moment solution regarding the QuQuickBooks Enterprise Support Number to the customers. Our technically skilled professionals are well regarded for smart technical assistance around the world.

    ReplyDelete
  47. We have the best plus the most convenient solution to boost your productivity by solving every issue you face with all the software. Give us a call at QuickBooks Phone Number to avail the greatest customer care services made for you.

    ReplyDelete
  48. You need a hand for support. QuickBooks payroll support is an answer. If you wish to accomplish this through QuickBooks, you receive several advantages. Today, payroll running is currently complex. You may need advanced software. There must be a premier mix solution. QuickBooks Payroll Support Phone Number often helps.

    ReplyDelete
  49. Getting instant and effective help for any question of concern is exactly what the user’s desire for. With QuickBooks Enterprise Technical Support Number, you are able to rest assured about getting the most desirable and efficacious help on every issue which you might encounter yourself with.

    ReplyDelete
  50. QuickBooks Pro is some type of class accounting software which has benefited its customers with various accounting services. It offers brought ease to you by enabling some extra ordinary features and also at Technical Support QuickBooks Phone Number telephone number it is simple to seek optimal solutions if any error hinders your work. With QuickBooks Pro you can easily easily effortlessly create invoices and keep close tabs on every little thing like exacltly what the shoppers bought, just how much they paid etc. In addition it lets you have a crystal-clear insight of your business that can help someone to monitor your cash, taxes as well as sales report, everything at one place.

    ReplyDelete
  51. A QuickBooks Payroll Tech Support Number is a type of subscription this is certainly done to activate the payroll features in your QuickBooks Desktop Software. Intuit Online Payroll exports transactions to QuickBooks Desktop along with Quickbooks Online as standalone software.

    ReplyDelete
  52. The name associated with the software is QuickBook Support Phone Number. QuickBooks, particularly, does not need any introduction for itself. But person who is unknown for this great accounting software, you want you to give it a try.

    ReplyDelete
  53. Creating a set-up checklist for payment in both desktop & online versions is an essential task which should be shown to every QuickBooks user. Hope, you liked your site. If any method or technology you can not understand, if so your better choice is which will make call us at our Quickbooks Enhanced Payroll Customer Support.

    ReplyDelete
  54. HP printer not connecting to computer difficulty can certainly be precluded by making one system as a number and letting the HP Printer Tech Support other system share the connectivity.

    ReplyDelete
  55. With QuickBooks Payroll Support Number it is possible to fill all the Federal forms by simply few clicks and send them to your appropriate federal agencies. QuickBooks payroll assisted, intuit files and pays the taxes for your requirements.

    ReplyDelete

  56. Amazing Features of QuickBooks Payroll
    The QuickBooks Payroll has its own awesome features that are good enough when it comes to small and middle sized business. QuickBooks Payroll also offers a dedicated accounting package which include specialized features for accountants also. You can quickly all from the QuickBooks Payroll 24/7 Support Number to learn more details. Let’s see many of the options that are included with QuickBooks that features made the QuickBooks payroll service a big success.

    ReplyDelete
  57. Printer is growing high with its broad range of customers. The latest version of printer inventory measures are just amazing, and with different models, for you to deal with. For matching the various mindset of people, both wired, as well as wireless printers, are now available from retail outlets and online stores. It is always advisable to come in direct contact with the original manufacturing houses while buying a piece
    HP Printer Support Phone Number

    HP Printer Support Phone Number

    HP Printer Support Number

    HP Printer Support

    HP Envy Printer Support

    HP Printer Customer Support Number

    HP Printer Customer Support Phone Number

    HP Printer Helpline Number

    ReplyDelete
  58. Nowadays, QuickBooks Support Number has become an important accounting software for small and mid-size businesses. The seamless financial management process of this software gravitating plenty of businessman each and every day.

    ReplyDelete
  59. QuickBooks Payroll Support Number could be the toll-free quantity of where our skilled, experienced and responsible team are available 24*7 at your service. There are a selection of errors that pop up in QuickBooks Payroll which are taken care of by our highly knowledgeable and dedicated customer support executives.

    ReplyDelete
  60. Our QuickBooks Customer Tech Support Number channel- We comprehend the complexity and need making use of this accounting software in day to day life. You can’t look out for pretty much time for this to have a fix of the single QB error.

    ReplyDelete
  61. Entering your details each time you make any transaction takes your time and in addition opens windows of committing mistake specially when you're in hurry and so this amazing property of QuickBooks Support Number helps to make your work easy.

    ReplyDelete
  62. Though these features be seemingly extremely useful as well as in fact these are typically so, yet there are lots of loopholes that will trigger a couple of errors. These errors may be resolvable at QuickBooks Support Phone Number, by our supremely talented, dedicated and well-informed tech support team team.

    ReplyDelete
  63. The support specialist will identify the difficulty. The deep real cause is likely to be found out. All the clients are extremely satisfied with us. We have many businessmen who burn off our QuickBooks Support service.

    ReplyDelete