In Part 1, we saw how to build uber/fat jar for our flink application. In this tutorial, we will cover how to deploy the app in KDA.
- Kinesis Stream
- S3 Bucket
- IAM Role
Let's create a Kinesis stream for feeding our Flink Application. You can also use Kafka or RabbitMQ as a source.
Deployment of flink application jar in KDA requires an S3 bucket (to act as a repository). We upload our jar into this S3 bucket and then point the KDA source to S3 Object URL. Let name the S3 bucket as
We need an IAM role which has only the right amount of access for our streaming job. We can set policies to limit the application access to particular kinesis stream etc.
NOTE: An IAM role gets automatically created if you specify the same during KDA app creation in AWS Web Console.
Create a KDA Application
Via AWS CLI:
This option is really good if you want to automate the KDA app creation part. There were many times when I needed to delete the running KDA app and create a new app with the same configurations.
- Fill in the necessary details in the below KDA app-create-config JSON.
- Run the following command to create the KDA app.
- If you face any issue in the above code, you start fresh with the skeleton code as follows:
Via AWS Console (Web)
For creating a KDA app via the web, you can refer the following docs.
Once everything is set you can click on Run and you application will start running. You will find you applications Job graph in the KDA UI.
Since we used
.startNewChain(), we will see blocks representing each operator ( with configured
If you hadn’t used .startNewChain(), you would be seeing a monolith block clubbing all your operators.
You can see HASH/FORWARD/REBALANCE/BROADCAST on the operator arrows. What does that signify? You can read my answer on Stack Overflow.
You can find you application logs in the cloudwatch console (If previously enabled while creating KDA app). Search for the log group with prefix
The logs would be in JSON format. You can make it more legible by using
cloudwatch insight .
You can select your log group and use the below filter.
Similarly, you can create a dashboard for monitoring metrics for your KDA app.
I have covered some finding w.r.t KDA and flink, based on my experience using it. Hope this helps someone.
Found it Interesting?
Please show your support by 👏.