Global variables — wait on Data and application state using Oracle NoSQL
This best practice shows how to save variable state for applications in OCI NoSQL. This can be easily shared across your applications. Within data integration there is often the need to trigger processing with time and also with some data, when data or some data values arrive in a data store and then trigger some processing. There is also the need to persist and retrieve data in global variables, a practice used in application development since time began, not necessarily good at times but still a requirement.
Introduction
Application state can be easily stored in NoSQL, OCI NoSQL makes this very easy to use, we will use OCI NoSQL for this illustration. For the illustration the state is stored in a table with an id column and a data column of type JSON. The data producing application will insert / update a row in the NoSQL table — this can easily be done using a REST API in OCI.
There is a simple to use set of APIs and also its available in OCI Console;
Below you can see the OCI NoSQL definition when it is created;
For example inserting id with value 1 and the data value as {”status”: ”DOIT”}.
Use case 1 — Wait for Data to Arrive
Scenario here is that a job is triggered on a schedule at 10pm EST, there is a window from midnight to 6am EST where the processing may be done but it is triggered when data in a table is landed. This solution uses OCI NoSQL and REST Task, the table is defined in NoSQL and the REST Task polls retrieving the information from the NoSQL table until the specific value is there.
Below you can see the pipeline that queries the table and waits until data is ready and loading should be done.
The REST task Wait for Data from NoSQL shown below is used in the pipeline illustrated above.
This REST task has an API call on OCI NoSQL to get a single row;
and waits until a user specified JSON path expression and value is reached. Otherwise there is a timeout which has the window of the batch run.
The above wait is defined as an 8 hour window with a polling check every 1 minute, these are all configurable.
Use case 2 — Global Variables for Managing Application State
The pipeline below illustrates how state can be retrieved and stored in OCI NoSQL, it can be used in all the operators such as the decision operator shown below; if the variables stored in the state have a specific value take one path, otherwise take another. Or store the response from a request in NoSQL.
In the above data pipeline, the first REST call gets the current application state, this is then used in the decision operator to drive behavior, it could also be used to configure payloads for subsequent calls, the application state is then updated in one of the conditional flows.
Summary
This best practice showed how to save variable state for applications in OCI NoSQL. This can be easily shared across your applications. Check out the OCI Data Integration tutorials referenced below and the example postman collection which contains the tasks above. Would love to hear your comments, feedback and ideas!
References