OCI Data Integration REST Tasks — Postman Collection

6 min readJan 3, 2023

As much as I’d like to write about Pinball Blizzard in this fantastic pinball arcade (which was truly awesome — have never seen so many working pinball machines even in the craziest of storms), this post illustrates a sample Postman collection which defines a number of REST Tasks for OCI services, this covers many services from OCI Object Storage to the Notifications service, Compute, Container Instances, Data Safe, Data Science Jobs and a number of AI Services (Vision, Document Understanding, Speech and so on).

REST tasks are a great way of extending and integrating arbitrary services into your data pipelines — you can schedule them on a recurring basis and also call from pipelines describing dependency flows. They can be parameterized so are highly reusable — just like the library here; all the tasks are parameterized so you can invoke in any OCI region for example using your own data.

The link below includes;

a Postman environment “OCI Credentials” for defining how you authenticate with OCI from Postman
a Postman collection “Data Integration REST Tasks” — many OCI services from OCI Object Storage, Container Instances (easily run applications on serverless compute optimized for containers) to many AI services.
a Postman collection “Data Integration APIs” — execute tasks, publishes tasks to applications, copies applications and so on.

Check it out, there are lots of examples you can leverage, or take, edit and customize.

GitHub - davidallan/oci-di-postman: OCI Data Integration Postman Collections

This contains one Postman Environment for OCI and two collections with example of OCI Data Integration. Download the…

github.com

Up and running with Postman

Download the repository’s Postman collections and environment.

Import the environment into Postman using the ‘Import’ button at the top;

Then activate it by selecting it from the top right drop-down.

Open and Edit the newly imported environment, and set the OCI REST variables tenancy_ocid, user_ocid, fingerprint and private_key (and region).

Now import the two collections into Postman.

Configure the Collections

That’s mostly it! For each collection there are collection variables that must be set.

The collection “Data Integration REST Tasks” has the following variables you need to set; WORKSPACE_OCID — the workspace where you want to define the REST tasks, PROJECT_KEY — the project key to create the tasks in and REGION — the OCI region name to use. For this collection you can execute the collection and all REST tasks will be defined in the project. You will need to publish them to an application and setup policies (resource principal access to each service) before you can execute the tasks.

The collection “Data Integration APIs” has the following variables you need to set; WORKSPACE_OCID — the workspace where you want to work, APPLICATION_KEY — application to execute task from, TASK_KEY — task to execute, REGION — OCI region name to use. There are other variables for some of the other APIs such as PROJECT_KEY to publish from for those examples. With this collection you have to execute individual API examples (do not run the entire collection). There are other example APIs there if you want to know more let me know.

Now you can use the collections to invoke any OCI REST APIs. Just make sure that the OCI REST calls are executed as part of the collection, as that collection contains the necessary javascript code to generate OCI’s authentication header.

Data Integration REST Library

So what do you get get? Below my project is called REST Library, you can see all of the REST Tasks that have been defined — below there are examples of different services from OCI Object Storage to Data Safe.

On the next page there are tasks for executing commands on compute instances and many of the AI services — these are mainly the batch like APIs that process data in OCI Object Storage and store the results in OCI Object Storage.

All of the REST Tasks are parameterized so you can use in your environment. They all are synchronous, so they will execute and wait for completion. If you do not want to wait you will have to remove the polling from the tasks.

OCI Data Science Example

The OCI Data Science service job run is included in the collection, this lets you easily orchestrate your Data Science jobs from within OCI Data Integration. You can schedule the Data Science jobs and also add the tasks into pipelines and easily orchestrate with other activities.

The task has parameters for the region to use, the inputs and outputs.

You can see within the REST task in the request body, the Data Science project and job are parameters along with various other aspects to the data science job, you can use this to run any Data Science job.

I used the “hello world” example in the OCI Data Science tutorial here, you can take this and parameterize further if needed. When I run this from OCI Data Integration from the applications page I can see the REST task running;

You will then see from the Data Science jobs page, the job go through the different phases

Then finish in Succeeded state!

Now you have a task its easy to orchestrate many Data Science jobs and other work in data pipelines, the example below runs 2 data science jobs concurrently then calls some tasks to analyze the resultant work.

Let’s look at another example.

OCI Document Understanding Example

As well as your custom Data Science jobs, OCI also provides a number of AI services such as the OCI Document Understanding service, this has also been added in the AI Document Table Extraction Task

The task has parameters for the region to use, the inputs and outputs.

You can see within the REST task in the request body, the feature type is hard-wired (TABLE_EXTRACTION below), you can take this and parameterize further. I’m interested to know the cool things you can do with this as a basis.

With this task, you can extract data from unstructured information and get insights into information that is otherwise hidden using OCI Data Integration to extract information from the results of these AI services.

There’s plenty of other tasks in the collection, check it out!

Hope you found this useful, check out the documentation for OCI Data Integration below. Send me comments, questions and ideas, would love to hear from them — even if its on pinball, definitely a great place to hang out in the blizzard of the century in the US. Pinball Blizzard!

Related resources