Serverless Lambda function with Talend Jobs

/**/

Introduction

Here we will learn how to make a serverless project to build a lambda function which will run the talend jobs.

Prerequisites

Installation of Node JS

Download the Node JS binaries from (https://nodejs.org/en/) site and install.
Node JS installation will install both node JS runtime and npm (node package manager)
NPM is used to install packages.

For linux we have to use command as 

yum install-y gcc-c++make
curl -sL https://rpm.nodesource.com/setup_6.x |sudo-Ebash
yum install nodejs
node -v

For windows we have to install like : https://nodesource.com/blog/installing-nodejs-tutorial-windows/

Installation of Apache Maven:

Following are the ways how we can install apache maven :

In Windows :

  • Go to the link :https://maven.apache.org/download.cgi unzip it and add the bin path to the environment variables like :
    • PATH=C:\apache-maven-3.6.0-bin\apache-maven-3.6.0\bin
    • We can verify if it is installed by checking mvn -version [Which will give us the version of the maven which is installed.]

In Linux :

  • wget http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz

  • Run the wget command from the dir you want to extract maven too.

  • run the following to extract the tar,
    tar xvf apache-maven-3.0.5-bin.tar.gz

  • move maven to /usr/local/apache-maven
    mv apache-maven-3.0.5  /usr/local/apache-maven

  • Next add the env variables to your ~/.bashrc file

    export M2_HOME=/usr/local/apache-maven
    export M2=$M2_HOME/bin
    export PATH=$M2:$PATH

  • Execute these commands
    source ~/.bashrc
  • Verify everything is working with the following command

        mvn -version

Installation of Oracle JDK

Serverless configuration

Reference:https://serverless.com/framework/docs/providers/aws/guide/quick-start/

  1. Run the below command to install Serverless globally

    serverless-stack-output is a plugin, aws-sdk is used to call Batch jobs and to install other dependencies using npm install

    npm install -g serverless
    npm install
  2. serverless config credentials

    sudo serverless config credentials --provider aws --key <ACCESS_KEY> --secret <SECRET_KEY>

Serverless installation of the AWS Lambda function

  • First we will make empty directory named as : serverless-lambda-talendjobname
  • Now in Visual code we will browse to the folder as cd serverless-lambda-talendjobname
  • Now we will create a serverless java maven template as : 
    • serverless create --template aws-java-maven
  • Now we have to install the talend related directories for that we have one zip attached as Supporting_Talend_Jobs_For Serverless
  • This zip will help us to install all the lib files and will also generate the pom.xml for us which we can use it for the serverless_Project.
  • Following will be the generic folder structure which we will follow :
  • Following are the details of all the folders and zips present in the above image
    • serverless_project : We can give the generic name to the serverless_project which will be parent directory and where we will place all the related folders and zips .
    • code : This will contain the serverless code which we will install and deploy
    • lib : This will contain all the libs which are present and which needs to do mvn install
    • Supporting_Talend_Job_For_Serverless : This is the attached zip which is used to install the libs of the talend projects
    • TalendProject : This is the talend project for which we have to create the lambda function
  •  Working Procedure of Supporting_Talend_Job_For_Serverless :
    • 1st we have to unzip the Supporting_Talend_Job_For_Serverless and also unzip the TalendProject
    • TalendProject have jars at 2 places : 
      1. TalendProject\lib
      2. TalendProject\TalendProject\lib
      3. We will place this jars in the lib folder which is shown in above image
    • Then we will run the bat file present in the directory D:\serverless_project\Supporting_Talend_Job_For_Serverless_0.1\Supporting_Talend_Job_For_Serverless\Supporting_Talend_Job_For_Serverless_run.bat which will do all the installation of the jars and also will generate the pom.xml file named as pom_generated_by_talend.xml
  • We will now replace the pom.xml file created in the code directory from the template to the one which is generated by talend
  • We will change the Handler.java as per the code like below
    • public ApiGatewayResponse handleRequest(Map<String, Object> input, Context context) {
      LOG.info("received: {}", input);
      testproject.newjob_0_1.newjob t2 = new testproject.newjob_0_1.newjob();
      String[] context2 ={};
      t2.runJob(context2);
      Response responseBody = new Response("Success",input);
      return ApiGatewayResponse.builder()
      .setStatusCode(200)
      .setObjectBody(responseBody)
      .setHeaders(Collections.singletonMap("X-Powered-By", "AWS Lambda & serverless"))
      .build();
      }
  • Now we will try to install the dependencies as :

    mvn clean install
  • When the build will be sucess we will deploy the sls as : 

    sls deploy
  • We can now check the function by invoking it as :
    serverless invoke --function <functionname> -l

Tips and Tricks

  • While developing we may need to again and again deploy the function which is difficult as it takes time to deploy using sls deploy
    • So we can deploy a specific function like : sls deploy –function functionname

Appendix :

Reference links :