Getting Notification of Data Integration Task Executions

In this blog post we will see how to get notifications for OCI Data Integration task executions. Each time a job is started or completed we can post use the OCI Events service to post a notification, let’s see the completion notify then we will see how to get notify in error cases.

Firstly I have created a topic in the notification service. The topic name is TasksCompleted, I have also created a subscription which uses email — when you add a subscription you will get an email accepting the subscription. You can then test using the “Publish Message” button on the topic panel in OCI Console;

Create a notification and email subscriber

Next we will create a rule, the rule will match on Data Integration task completion, when any task completes, the action will be to post to the notification above.

Below we define the rule — the condition is for service Data Integration and event type of Execute Task — End (there is also an event type of Task Completion — Exceeded Expected Duration). The action defined below is to post a message to the topic we created above.

Create a rule to match on task end

When we execute a task, each time we will get a standard OCI message with the title “OCI Event Notification :com.oraclecloud.dataintegration.createtaskrun.end” and the body will contain the JSON of the event — which will include the task name and the task key.

Notifying on error

This can be done ootb using an additional attribute filter for Task Status with value ERROR, see this gif for an example;

Easy and you can get email, pagerduty, SMS messages and so on

Notify on error — custom message

If we wanted to notify on error and provide some custom message / information, today you can create an OCI Function that acts as the rule action and then checks the result of the task run to see if it is successful, if it is not successful, you can post a notification with a user-friendly body including the error information.

The code is below;

With this function deployed you can then configure the runtime configuration values as below, this includes the workspace and topic to target.

fn config function YOUR_FN_APP_NAME notifyonerror workspace YOUR_WORSKPACE_OCID

fn config function YOUR_FN_APP_NAME notifyonerror application YOUR_APPLICATION_KEY

fn config function YOUR_FN_APP_NAME notifyonerror topic YOUR_TOPIC

fn config function YOUR_FN_APP_NAME notifyonerror region YOUR_REGION

The function will check the result of the task run and post a notification, the email will look something like this containing the error message from the task along with a link to the OCI Console to the task run;

Example email with error

The link in the email will open the OCI Console in the data integration execution run;

You can then see the parameters, view the errors and so on. The summary error is the same one we saw in the email;

Notify on Task Exceeding Expected Duration

There’s a built in event type for tasks that exceed the expected duration, if you use this you can be alerted for tasks that exceed a specific duration configured when you define the task schedule.

Summary

You can see how easily it is to be alerted on error when executing data integration tasks and given information that will help you get to the bottom of issues. This is an example of using the event service to get notified of activity in data integration service, I hope you found it useful.