Realtime Response using Talend ESB

In ETL, there are scenarios when data needs to be processed as soon as client upload files into the system. Generally, we use triggers available in Data integration tools for such scenarios like wait for a file or we will trigger process in every 5 mins to check for the new file and process it. We always try to achieve realtime in loading data to our system.

Data integration tools do not provide much capabilities to respond to the client in realtime. It can provide fast and efficient way to process and validate data, but to give response to client in case of validation failure and not allowing them to upload wrong files by providing error messages in realtime can be achieved from service based system.

I have been facing same problem from 3 mins gap to reach realtime in loading data and to give response back to client in realtime if he commit any mistake in uploading.

I was using Talend Open Studio for Data Integration but to reach in realtime I shifted to Talend ESB v.6.1.1. Talend ESB studio has two prospective Integration and Mediation. Integration is same as from Data Integration studio and Talend ESB also comes with Runtime.

Screenshot 2016-01-24 03.15.23.png

As both Data Integration studio and ESB studio support same components for data Integration I did not need to do any re-design for the logic in reading and writing data. Now in ESB I just need to wrap my ETL logic inside a RESTful web service which can be called from web browser with client specific parameters and can return response back in realtime.

Screenshot 2016-01-24 03.35.32.png

Above snapshot has the request made to the dummy RESTful web service MyFirstAPI using Talend ESB.


Screenshot 2016-01-24 03.57.40.png

In MyFirstAPI job I have used tRESTRequest to receive a request from browser using url pattern “talend/{action}” where action can be anything but job will return valid status only when action is valid. And using tFlowToIterate we can iterate request from clients and perform ETL specific to clients or actions.

Please follow Talend ESB RESTful Service  and Deploy Talend ESB Service to know how to create and deploy RESTful Service.

Please let me know your views I hope the above is useful to you.

3 Comments Add yours

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s