Lifecycle Management with Applications in OCI Data Integration
Keeping your data integration applications in order during their lifecycle is straightforward with OCI Data Integration, generally development starts and projects get created containing your integration tasks, these are published into applications (the publish step is done by creating a patch containing tasks to be published and the result is published objects) and development tests run (see this blog on that process). For more formal testing a separate test application can be created based on your development application — this can be in the same workspace or separate workspace as development (it can also be in a different tenancy entirely — application developed/tested in one tenancy and published into a production tenancy). In this application you can map the data assets to your test data assets, or use parameters in parameterized tasks to accomplish this (this is the best practice for doing this) or reuse the development ones if its in the same workspace. Any changes in the development application can be kept in sync, this can then be copied to production by again creating a production application and then mapping the data assets, synchronizing content and so on (rinse and repeat). The image below illustrates that flow.
Let’s look at this in action. Below we see our development project we will use in this illustration, we can select the tasks to publish and click on “Publish to Application”.
A dialog will appear ask for the target application to publish the objects to.
When you click on publish, you can view the published tasks in the application, the patches tab will let you monitor progress if there are lots of content being published.
At this point you can test your work, execute the tasks, ensure they are functioning correctly and so on. When you are happy with the state of this, create an application for QA/testing. This is done by creating an application in the workspace you are using for your testing — this could be the same workspace as your development or in a different workspace entirely as illustrated in earlier image. The create application dialog has an option “Copy resources from an existing application” it is this option you can use to copy from dev to test, test to production and so on. Here you can also copy an application from another tenancy (perhaps a development tenancy) to this tenancy (a production or test tenancy). Select the source tenancy ocid, the compartment/workspace and application.
You can now view the created application for our testing and see the tasks, references and patches. Notice the source application information and tooltip displayed below which was not on our development application — theres also a very important sync link.
Clicking on ‘View Details’ you can see the full info such as the compartment/workspace and application name.
If there was more work that happened in development you can synchronize the content from the development application by clicking “Sync”. This will synchronize the application with the development application, in the dialog you can view updates/additions/deletions and review the changes.
Let’s look at the patches and we can see that these actions have all been recorded and there is information in the patch that defines exactly what was done to my application. I can drill into the patch to see this with even more detail or if there was some issue.
Earlier I mentioned that you may have different data assets in testing, most definitely in production, the best practice way for handling this would be to use parameterized tasks then you can just execute and pass the data asset you are using — option 1 below. You can also remap data assets as in option 2 below.
Option 1 — Use Parameterized Tasks
See Aditya’s blog on using parameterized tasks and passing different data assets for dev versus test versus production;
Option 2 — Remap Data Assets
It also possible to remap what was configured in the source application with what you want to configure in this one to test/run with is to view the ‘References’ tab. Here you can see all of the data assets referenced in the application AND change the mapping of what there were to your test or production environment.
In above ‘References’ table, I have clicked on ‘Map‘ in the mapped to column for my data asset, I will map it to a different data asset for my testing work. You should ensure this represents the same data asset but be a test or production variant.
Below you can see that this is now mapped and that the connection can now also be mapped…
You can also see where the connections and data assets are used. For example below you can click on the right side of the row and see the options to ‘Map’ and ‘Used In’.
The ‘Used In’ option will indicate the tasks where the connection or data asset is used;
One tip for when using Fusion / ATP / ADW data asset types is for how to set the default staging location — don’t set the default staging location property in the dataflow source or target operator — use the data asset default staging location. You can’t remap that property — so to be successful for that you will need to use the default staging location in the data asset;
Within the dataflow use the data asset default staging location, don’t set this to a specific one;
As you can see, here we can promote applications through the lifecycle, manage the references to reference different environments (dev/test/prod) and also understand where such information is used.
After going through the above steps, we see now the connection can be mapped to a new environment, we can understand which tasks use it and we are complete. We are ready to test! When all is good with testing, we can go through a similar cycle in creating an application for production, ensuring its synchronized with any changes and mapping/using the data assets and connections for that environment.
The same copy/sync functionality for applications works within a workspace also, so it’s possible to carry out the development to test etc within it and also have different data assets to cleanly separate your development and tests like this;
Hope you found this useful its a very quick flyover of creating applications and synchronizing content from development through testing and production.
Check out all the blogs related to Oracle Cloud Infrastructure Data Integration — https://blogs.oracle.com/dataintegration/oracle-cloud-infrastructure-data-integration. To learn more, check out the Oracle Cloud Infrastructure Data Integration Tutorials and Documentation.