ODI 11g, ODI 12c : What’s an Agent?

What is an agent?

An agent is a java process which usually located on server and listens to a port for incoming requests. It runs the requested scenario, reverse engineers requested datastores etc.

When a job submitted through ODI Studio GUI or through startscen.sh agent gets scenario from work repository and topology definitions from master repository, combines and converts them into a runnable job, usually consisting more than one code block. Then it sends code blocks to destination environments, which may be DB servers, file servers, hadoop name node etc. Finally agent gets job statuses from these environments and writes into work repository tables for us to see from Operator tab of ODI Studio.

Agent Types

Standalone Agent

It is the basic agent of ODI. It does not require an application server like JEE Agent. It is easy to configure/start/stop this agent from shell. Since today I’ve always used this agent and never tried other versions. A-Team article says this is the most light weight and low footprint choice.

JEE Agent

“Java Enterprise Edition” agent, which requires an application server, in most documentation you can see the name of WebLogic Server since it’s another Oracle product. (Some search results: depending on this CertMatrix of Oracle http://www.oracle.com/technetwork/middleware/data-integration/odi-11gr1certmatrix-ps6-1928216.xls ODI 11.1.1.7.0 only supports WLS and does not support Tomcat or other application servers. You may -or may not- configure them to run together but it is not supported.) This agent is first delivered with ODI 11g. And still exists in ODI 12c.

Some pros of JEE agent which are written in A-Team blog are:

High availability : Through Web Logic Server’s cluster architecture, even a node is down agents may run on other nodes.
Configurable connection pooling : Connection pool can be configured through WLS.
Monitoring : Oracle Enterprise Manager can monitor, configure, alert, manage ODI JEE Agents. But I believe, there is a plug-in to be installed to achieve this tasks from OEM.

Colocated Agent

The newest agent type which has arrived with ODI 12c. This type is like a combo of other two types. Agent is a standalone agent, but can be monitored and configured through WLS. Unfortunately it does not take advantage of connection pooling, high availability. Our agent will be in WLS domain, can be managed through WLS and that’s all. It is lighter than JEE Agent. In my opinion companies which prefers JEE Agent as production agents can choose colocated agent as their DEV/TEST agent.

Agent Types diagram from Gerard Nico's website — Agent Types diagram from Gerard Nico’s website

Where to locate an agent?

To decrease network I/O it is better to locate agent to target DB server. Since agent submits code to DB Engine, it is better for them to be on same machine. Don’t forget that ODI is an E-LT tool. Which means it will load data into target server, then it will transform your data. So most of the load will be on target server. Which also means most of the codes will be submitted to target server.

Also since an agent is a local java process, agent will write files to the machine, which it is set-up on. If you have a file server seperate than DB server, then it is better to have another agent on file server to handle read/write file processes. Or to mount file servers directory to DB server as a directory and setting up only one agent is another solution.

Also these solutions will prevent any firewall related problems.

Thanks for reading, don’t forget to share & comment.

Resources I’ve read before writing this post:
http://www.ateam-oracle.com/odi-agents-standalone-jee-and-colocated/
http://www.ateam-oracle.com/understanding-where-to-install-the-odi-standalone-agent/
http://odibeginnertutorials.blogspot.com.tr/2013/11/2-introduction-to-odi-agents-and.html
http://www.odigurus.com/2011/10/defining-odi-agents.html
http://gerardnico.com/wiki/dit/odi/agent

ODI 12c: First Look and Repository Creation

Hello,

After a long pause on blog, here I am again. Oracle Data Integrator 12c is finally available for everyone to download. So in this post I will discuss about my first impressions and I will explain how to create repositories, both master and work. Actually it is pretty simple and almost same with 11g which I told in this post.

So first impressions, when you download ODI 12c through this page, you will get odi_121200.jar (numbers can differ with time since it’s version number) and some opatches bundled with it. Actually it is a bit disturbing for me to have a jar file which is 1.8GB. I’d like to have an exe for Windows.

Anyway I had some problems with running this jar also, first I tried it on my VM which has 32bit Windows 7 and got an error that states it could not reach jar file. So I moved to my physical machine that is 64bit Windows 7 and OUI could not recognize the platform and exited everytime, until I download and install Java 1.7. So after solving the problem with Java, I moved to my VM again to solve other problem where it came out that my path is problematic, since my user name is Canburak Tümer, space created a problem to reach file. I created another user without space that can run the installer.

Finally I could see the installer UI. It was a pretty straight forward installation, I just selected enterprise and went on. After installation, I ran the ODI Studio, it has a really clean and elegant splash screen and it asks to migrate any user settings from old installations. After splash screen, ODI workbench has been load:

Creating Master Repository

As I mentioned before, repository creations are almost same with ODI 11g, we will start by clicking File > New and we will see screen below:

Select “Master Repository Creation Wizard” in ODI tab and click “OK”.

You will see screen above, where we will enter database information, schema where we will create repository and DBA user to run some of the creation scripts.

Define and confirm password for SUPERVISOR user. DO NOT FORGET THIS PASSWORD UNLESS YOU HAVE ANOTHER USER WITH SUPERVISOR PRIVILEGES. For this reason many ODI developers/admins make this password “SUNOPSIS” as an old habit. I prefer to have it as “SUPERVISOR” in my VM and personal development environment.

Select password storage as you wish. Then click finish, it will run scripts now to create master repository, it took around 4 minutes in my VM, probably it will take around 2-3 minutes in your physical machines. Now it’s time to create a connection to master repository.

Click on “Connect to Repository” then click to green plus in the pop-up window, then fill required information in the form. Use SUPERVISOR as ODI user and DB user which you have created the master repository with. Make sure you have selected “Master repository only” radio button. Then click “OK”.

ODI 12c will ask you to if you want to keep passwords in a secure wallet with a master password. I do not have enough information about this wallet yet, but I will learn and write another post about it. I prefer the less secure way which does not include the wallet. Now we have master repository and connection to master repository. Now it’s time to create work repository.

Creating Work Repository

To create work repository, connect your master repository then go to Topology tab and expand Repositories menu.

Right click to “Work Repositories” and click to “New Work Repository” from menu.

Insert connection information of schema which you want to create Work Repository in. (I had a problem with this step, actually I wanted to use odiw_c user but ODI 12c keeps filling the form in upper case so it gives invalid credentials error.)

In final step, insert repository name and select repository type.

You can also define a password for repository, which is different from ODI user password or DB user password. This password is just to secure the repository connection. When you click “Finish” it will run scripts to create work repository and will ask you if you want to create a connection to work repository. It will create a connection without ODI user information. So you will need to edit connection to insert ODI user information.

After all these steps we have installed ODI 12c and setup both master and work repositories for our environment. And we have a final view as below :

Now, it’s time to create our topology connections, models, projects; import or reverse engineer data sources; develop mappings (new name for interfaces), packages and more.

Welcome to ODI 12c, keep following my blog for further posts and please do not hesitate to contact me through comment form below.

ODI 11g : Step by Step Master and Work Repository Creation

Hello all,

So first post, first steps. We are going to create master and work repository for your Oracle Data Integrator. I assume that you have already have an ODI installation up and running and also an Oracle Database.

First I am going to create schemas for my repositories, you can put your repositories in one schema but it is common to have them in separate schemas. If you run the script below you will have two schemas for your repositories with all privileges.

create user ODIM_BLOG identified by ODIM_BLOG;
create user ODIW_BLOG identified by ODIW_BLOG;
grant all privileges to ODIM_BLOG;
grant all privileges to ODIW_BLOG;

Creating Master Repository

So if your schemas are ready, we can continue with the master repository. In the main ODI screen, go to File > New.

Now you’ll see three options, we’ll continue with Master Repository Creation Wizard, select it and click OK to continue.

Now it is time to setup our connection to database, you will need DBA connection to setup your repository. Fill in the required fields, you can see my blog environment’s information below. A little caution here: if you are going to use SYS user as DBA account, you have to fill DBA User area like ‘SYS as sysdba’ otherwise it will not connect. If you are using another DBA account then you don’t need to declare ‘as sysdba’. Hit the Test Connection button to see if you can connect.

There is another text box which asks for the repository ID, be sure to remember this ID, as you’ll also be asked for another ID while creating work repository. And it is important that all repositories have different IDs. As ODI does every operation with ID’s, it’s crucial not to have duplicated IDs.

Next step is the generating a password for SUPERVISOR user, this user will be our ODI admin until you will create another user with SUPERVISOR privileges. It is an old tradition to set this user’s password as SUNOPSIS, it is also a tribute like behavior for ODI’s creator Sunopsis. I set it as SUPERVISOR to remember it easily.

Here it goes, now we will wait until ODI finishes its job. If you check your master repository schema you can see it is being populated by SNP_% tables. SNP prefix is also a heritage from Sunopsis.

Master-5 — Master Repository Creation Step 5

And here we have created our master repository. Now it is time to setup a connection and create a work repository.

Master-6 — Master Repository Creation Step 6

Creating a Master Repository Connection

So for connecting master repository, click Connect to a Repository in ODI Home Screen. A little window will pop-up. This is the window where you can select between your predefined connection, edit them or create new connection. As we do not have any predefined connection, we will create one from scratch. Click on +(plus) sign to add a new connection.

MasterConnect-1 — Master Repository Connection Step 1

Now this is the connection details window. First blank will be the alias for this connection so it should be something to remind you to which repository you are connecting. First login information is for ODI user we are going to use, which is the SUPERVISOR in our case. And the second login information is for the Master Repository’s database schema, this is ODIM_BLOG in our case.Make sure Master Repository only radio button is checked before testing the connection. Because we do not have any other repositories yet. Test your connection then connect your master repository with SUPERVISOR. It’s time to move a step further.

MasterConnect-2 — Master Repository Connection Step 2

Creating Work Repository

When you are connected to aster repository go into Topology view and extend the repositories accordion. Right click on Work Repositories and click on the New Work Repository on pop-up menu.

Enter your schema information, test your connection and move on. In our case we are going to use ODIW_BLOG schema as we want our repositories to be separated.

Give a name and ID to your work repository. DO NOT forget to give a different ID from your master repository’s ID. Select your work repository type in this step.

There is two types of work repositories: Development and Execution. Execution repository does not have a development ability and does not have an active Designer tab. It only holds, scenarios, load plans, sessions and topology information. Development repository has an active Designer tab, which adds this repository to hold, create, edit, delete ability of interfaces, packages, variables, scenarios, knowledge modules, model objects like tables and views.

We will get into deeper about development and execution repository in another post. So I will make it short here.

Now let ODI to finish creating work repository.

Now ODI will ask if you want it to create a connection for new created work repository.

Work-5 — Work Repository Creation Step 5

When you click Yes, it will ask for connection name, which we said it is connection alias. And when you click OK. It will create a connection.

Work-6 — Work Repository Creation Step 6

Connecting to Work Repository

WorkConnect-1 — Work Repository Connection Step 1

ODI has created a connection for us. When you disconnect your master repository. (Under ODI > Disconnect) You can connect to work repository, click on Connect to a Repository, select your work repository connection from dropdown list. Password field will be empty, enter your SUPERVISOR password and click connect.

As we do not have any other user yet, we will connect to work repository by SUPERVISOR user.

Thanks for your patience to finish reading. Next ODI post will be about creating a user in detail, and assigning profiles to a user.

If you have any questions, please use comment section to ask me.