Oracle Cloud Infrastructure Data Integration and Fn

Oracle Cloud Infrastructure Data Integration is an Oracle-managed service that provides extract, transform, and load (ETL) capabilities to target data lake and data mart use cases on Oracle Cloud Infrastructure (OCI). Oracle Functions is a fully managed, multi-tenant, highly scalable, on-demand, Functions-as-a-Service platform.

Here we will see how to use Functions when you want to focus on writing code to meet business needs; the example we will see here is the ‘hello world’ example for Data Integration(posted here), this example will be extended in subsequent posts illustrating integration with other services in OCI such as the Events Service and Notification Service. We will use the OCI Data Integration Python SDK to list workspaces in the root compartment, this is using a resource principal rather than the user principal from the earlier hello world post, I’ll then extend this example with a parameter which is the compartment to list workspaces within.

I used my laptop to create the Fn function and deploy, you can also do this from an instance in OCI (there are instructions for that here), my policies were as below, I had a dynamic group which basically allowed all function resource types access (ALL {resource.type = ‘fnfunc’)}, this can be customized to much finer grained authorization to be specific to a specific function, compartment etc.

Navigation from Developer Services through to Functions;

In the Functions home you will list and create applications. Applications can contain many functions. There are plenty of tutorials for Fn, so I will keep to the minimal info here.

Let’s create a new application, as well as the name we define the network that the function will use. Notice I also enabled logs to be written to ObjectStorage, this is very useful for debugging, any standard out/error that you write in the function can be viewed in ObjectStorage after.

After the application is created we can see it listed and in Active status;

I’ve already created a few functions in application below, we will see how to create a function later;

The “Logs” link above takes you to the ObjectStorage bucket that has logs for your Fn function executions, from here logs can be downloaded and inspected (invaluable for debugging);

Designing and Deploying a Function

The functions when deployed are deployed into the OCI Registry as docker images so you’ll need to be able to login to the registry, below I’m logging in to IAD, creating a context called discontext which uses my DEFAULT .oci profile, configures the compartment to use and the docker registry (distools).

docker login iad.ocir.io -u yournamespace/youruser
fn create context discontext --provider oracle
fn use context discontext
fn update context oracle.profile DEFAULT
fn update context oracle.compartment-id ocid1.compartment.oc1..aaaaaaaa45umzgitiab4o5rodo67dnjov5lbbzbp54a3rdhzdd3hfdcmpz4a
fn update context api-url https://functions.us-ashburn-1.oci.oraclecloud.com
fn update context registry iad.ocir.io/yournamespace/distools

Now, let’s create the hello world for Data Integration; list workspaces in a compartment;

fn init --runtime python list-workspaces
cd list-workspaces/

Change func.py to have the content below;

import io
import json
from fdk import response
import oci
from oci.data_integration.data_integration_client import DataIntegrationClient 
def handler(ctx, data: io.BytesIO=None):
  signer = oci.auth.signers.get_resource_principals_signer()
  resp = do(signer)
  return response.Response(
    ctx, response_data=json.dumps(resp),
    headers={"Content-Type": "application/json"} 
  ) 
def do(signer):
  client = DataIntegrationClient(config={}, signer=signer)
  try: 
    inst = client.list_workspaces(signer.compartment_id)
    inst = [[i.id, i.display_name] for i in inst.data]
  except Exception as e:
    inst = str(e)
  resp = {
   "workspaces": inst,
  } 
  return resp

You can now deploy your function to the application you have created.

fn -v deploy --app distools

Invoke the function from the command line;

fn invoke distools list-workspaces

You’ll get the workspaces listed on standard out if you have any in the root compartment, which is unlikely… so let’s look at specifying a specific compartment via the function’s standard input.

Designing and Deploying a Parameterized Fn Function

As an example of how to parameterize the function, we can add a compartment identifier as an input and then use it if supplied.

Change the func.py to have the following content, the data parameter in the handler is passed into the do function where we parse the JSON and use the attribute compartment-id if defined;

import io
import json
from fdk import response
import oci
from oci.data_integration.data_integration_client import DataIntegrationClient 
def handler(ctx, data: io.BytesIO=None):
  signer = oci.auth.signers.get_resource_principals_signer()
  resp = do(signer, data)
  return response.Response(
    ctx, response_data=json.dumps(resp),
    headers={"Content-Type": "application/json"}
  ) 
def do(signer, data):
  client = DataIntegrationClient(config={}, signer=signer)
  compid = signer.compartment_id
  try: 
    body = json.loads(data.getvalue())
    compid = body["compartment-id"]
  except Exception as e:
    compid = compid
  try: 
    inst = client.list_workspaces(compid)
    inst = [[i.id, i.display_name] for i in inst.data]
    print(inst)
  except Exception as e:
    inst = str(e)
    resp = {
      "workspaces": inst,
    } 
  return resp

With the above function I can now do some interesting stuff. I can invoke with no parameters as before or invoke and pass the compartment in a JSON document that will be the body referenced in code.

echo '{"compartment-id":"ocid1.compartment.oc1..aaaaaaaa45umzgitiab4o5rodo67dnjov5lbbzbp54a3rdhzdd3hfdcmcccc"}' | fn invoke distools list-workspaces

Now I can see my workspaces;

{“workspaces”: [[“ocid1.disworkspace.oc1.iad.anuwwwwwwwwpeeko4wwww”, “PP1ViaUIWorkspace”]]}

This opens up other fun opportunities such as invocation of tasks via events — event service or notification for example. More to come!! In the meantime, try out Data Integration service on Oracle Cloud Infrastructure or check the references below.

References:

https://www.oracle.com/webfolder/technetwork/tutorials/infographics/oci_faas_gettingstarted_quickview/functions_quickview_top/functions_quickview/index.html

https://www.oracle.com/webfolder/technetwork/tutorials/infographics/oci_functions_gettingstarted_quickview/functions_quickview_top/functions_quickview/index.html#

https://github.com/arodri202/oci-rp-list-instances

https://medium.com/@ionut.vladu.adrian/event-driven-infrastructure-as-code-with-oci-events-functions-and-python-mytechretreat-9279f61ad477

https://www.ateam-oracle.com/jumpstart-your-functions-as-a-service-journey-in-oci-with-cloud-shell

https://www.ateam-oracle.com/oci-increases-notifications-fan-out-by-adding-functions-support