One of my customers had a high turnover rate. Sometimes we ran into some ODI objects which are locked by a developer who has gone. That triggered a question. What if I delete an ODI user while it has locks on objects. So I gave it a try, and while trying this other cases also pop-up. I also tried them on my VM.
Case 1: Deleting a user with locked objects
My expectation was being unable to delete a user. I was expecting there should be a FK between user and lock tables in repository. But I was able to delete. Then I thought about two scenarios,
a) object will be unlocked
b) object will stay locked and when I try to visualize lock, ODI will crash since there will be nothing returned from lock query.
I was wrong. You may find the screenshot about how ODI responds to deleting a user with locks.
I can open object, ODI will pop-up “This object is locked by DEVELOPER. You can not edit” dialog. You can view everything in the object. But to begin using object again, you need supervisor to unlock it.
Case 2: Deleting a user who created some objects
In that case I was expecting to find “Created By:” text box in object’s version tab to be empty. But I was wrong again. ODI just worked fine. When I checked the repository, I saw that ODI holds usernames for these text box not the user ID’s.
Case 3: Deleting an online user
After first two cases and failures of my expectations, now I had nothing to expect. I would just try and see. I created a user, open another ODI instance, connect with new user and deleted the user. Then I tried to take some actions in ODI. Some I was able to, some I was not.
Things I was able to:
View operator logs
View topology definitions
Things I was not able to :
Run interfaces or scenarios
Selective Reverse Engineering a model
Edit topology definitions.
If you have some other cases for me to try please write in the comments area.
Thanks for reading, dont forget to share & comment.
An agent is a java process which usually located on server and listens to a port for incoming requests. It runs the requested scenario, reverse engineers requested datastores etc.
When a job submitted through ODI Studio GUI or through startscen.sh agent gets scenario from work repository and topology definitions from master repository, combines and converts them into a runnable job, usually consisting more than one code block. Then it sends code blocks to destination environments, which may be DB servers, file servers, hadoop name node etc. Finally agent gets job statuses from these environments and writes into work repository tables for us to see from Operator tab of ODI Studio.
It is the basic agent of ODI. It does not require an application server like JEE Agent. It is easy to configure/start/stop this agent from shell. Since today I’ve always used this agent and never tried other versions. A-Team article says this is the most light weight and low footprint choice.
“Java Enterprise Edition” agent, which requires an application server, in most documentation you can see the name of WebLogic Server since it’s another Oracle product. (Some search results: depending on this CertMatrix of Oracle http://www.oracle.com/technetwork/middleware/data-integration/odi-11gr1certmatrix-ps6-1928216.xls ODI 184.108.40.206.0 only supports WLS and does not support Tomcat or other application servers. You may -or may not- configure them to run together but it is not supported.) This agent is first delivered with ODI 11g. And still exists in ODI 12c.
Some pros of JEE agent which are written in A-Team blog are:
High availability : Through Web Logic Server’s cluster architecture, even a node is down agents may run on other nodes.
Configurable connection pooling : Connection pool can be configured through WLS.
Monitoring : Oracle Enterprise Manager can monitor, configure, alert, manage ODI JEE Agents. But I believe, there is a plug-in to be installed to achieve this tasks from OEM.
The newest agent type which has arrived with ODI 12c. This type is like a combo of other two types. Agent is a standalone agent, but can be monitored and configured through WLS. Unfortunately it does not take advantage of connection pooling, high availability. Our agent will be in WLS domain, can be managed through WLS and that’s all. It is lighter than JEE Agent. In my opinion companies which prefers JEE Agent as production agents can choose colocated agent as their DEV/TEST agent.
Where to locate an agent?
To decrease network I/O it is better to locate agent to target DB server. Since agent submits code to DB Engine, it is better for them to be on same machine. Don’t forget that ODI is an E-LT tool. Which means it will load data into target server, then it will transform your data. So most of the load will be on target server. Which also means most of the codes will be submitted to target server.
Also since an agent is a local java process, agent will write files to the machine, which it is set-up on. If you have a file server seperate than DB server, then it is better to have another agent on file server to handle read/write file processes. Or to mount file servers directory to DB server as a directory and setting up only one agent is another solution.
Also these solutions will prevent any firewall related problems.
Thanks for reading, don’t forget to share & comment.
Warning! This post is written by using ODI 12.1.2, in version ODI 12.1.3 there are changes in agent setup. A post about ODI 12.1.3 will be written soon.
In this post I am going to tell you how you can set up a standalone agent for ODI 12c. Oracle provided a new tool for configuring our ODI 12c setup elements, such as WLS, Console, Agent, RAC. But to use this configuration manager you will need to create some schemas for OPSS, AUDIT, AUDIT VIEWER and luckily we have RCU for this task, you may do it manual also. But I prefer automation and I will describe in this post how you do it with RCU.
Important! Notice that even if you have a database which can work with ODI 12c, it can be unsuitable with RCU. I faced this problem, I had Oracle 220.127.116.11 for my ODI repository but RCU needed a newer version, so I installed ODI 12c database.
Remember! The paths I will give in this post are absolute paths for my installations, your path may change depending on where you have installed ODI 12c.
Shortly we are going to do
Create required repository and schemas by using RCU
Create a domain to run agent by using Configuration Manager
Define physical agent on ODI Studio
Test it from ODI Studio
1. Creating necessary schemas
You will find RCU tool in /home/oracle/Oracle/Middleware/Oracle_Home/oracle_common/bin for Linux OS and C:OracleMiddlewareOracle_Homeoracle_commonbin for Windows OS. File name is RCU.cmd or RCU.sh depending on OS. Start the RCU, select “Create Repository” and continue.
Fill all the spaces for the database you want to connect, be sure to enter full service name. In screenshot you will see it is filled as “orcl122” but full service name is “orcl122.localdomain”. I took the screenshot a bit early. Don’t let it trick you.
You will need to select schemas for RCU to create, Oracle Platform Security Services, Audit Services, Audit Services Viewer must be selected. When you check Audit Services, Audit Services Viewer and Audit Services Append will be checked automatically. Also in this step you can choose RCU to create Master and Work Repository for you. Since I have already created my repositories long ago I will skip it.
Set passwords for schemas that are going to be created. I will choose to set same password for every schema.
Set tablespace for schemas. If you leave it default, RCU will name them with a prefix which is set on third step. If tablespaces does not exist RCU will create them for you.
Now RCU will show you a summary of settings, check the list if it seem fine click on “Create”. It will take some time to create schemas, tablespaces, tables then it will show you a report. And you are ready to close RCU and continue with Configuration Manager.
2. Creating Domain for Agent with Configuration Manager
Start Configuration Manager, you can find it in /home/oracle/Oracle/Middleware/Oracle_Home/oracle_common/common/bin on Linux OS and in C:OracleMiddlewareOracle_Homeoracle_commoncommonbin in Windows OS. File name is config.sh or config.cmd depending on OS. Select “Create a new domain” and select where you want to put domain’s files. I will leave it as it is.
Next page is Template Selection page, we will need to check “Oracle Data Integrator – Standalone Collocated Agent” and when you check this option there will be two more options checked automatically.
Next step will want you to create a login for WLS. Password should be at least eight characters, including at least one letter and one number.
Select Domain Mode and JDK, I will leave them default.
Now we will select RCU Data option, since we created necessary schemas with RCU. Enter connection information for the database schema where RCU worked on before. Then click on “Get RCU Configuration” and wait for Configuration Manager to read from database. Then click “Next” when it is available.
Enter all passwords and schemas we have created before. Also you will need to fill database connection information again.
Configuration Manager will check for connections if they are working.
Now we need to select components which we are going to need, In this case we will only select “System Components” and “Deployments and Services” and continue.
Set name for the agent I will use OracleDemoAgent for this demo.
Next select the component from dropdown list, which we have created one step ago. Set listener information you can leave port as 21910 it is default for configuration manager. Fill in supervisor user information. And continue, for this demo we will leave settings default on other steps then click on “Create” on last step.
Creation will take sometime after that click on “Next” and “Finish” and we are almost ready to run the agent.
3. Defining Physical Agent in ODI Studio
Run ODI Studio, and login to repository, where you have Topology rights. We will need this rights to create agents.
Go to Topology tab, in Physical Architecture accordion, right click on “Agents” and click on “New Agent” a new window will appear on right panel.
Name the agent as the same name as we selected in Configuration Manager for the Agent component. When we try to run agent, it will search for its name in Master Repository so names should match exactly. Set the port as you set in Configuration Manager. Save the agent.
4. Start up the agent
Open a terminal/command line window and go to the domain folder that you have created. Under this folder you will find bin folder. For me this folders path is /home/oracle/Oracle/Middleware/Oracle_Home/user_projects/domains/demo_domain/bin in Linux OS or C:OracleMiddlewareOracle_Homeuser_projectsdomainsdemo_domainbin in Windows OS. Run the command agent with NAME parameter as following, or as you can see in screenshot. agent -NAME=OracleDemoAgent
Agent will start to rise up, it will connect to Master Repository, check for its name, read other configurations and start the service and listener. When agent is fully started, command line will seem like screenshot below.
5. Test the agent from ODI Studio
Now we have the agent up and running, also we have a definition for it in our Master Repository and a GUI in ODI Studio. It’s time to test if ODI Studio can connect to agent service. Go to Topology tab and open the agent we have created. There you will see a button written “Test” click on it. If everything is right until here you should see the screen below.
We have completed everything now. Agent is up and ready to run your jobs on databases for you.
Thanks for reading and please do not hesitate to write your comments or ask questions in section below.
After a long pause on blog, here I am again. Oracle Data Integrator 12c is finally available for everyone to download. So in this post I will discuss about my first impressions and I will explain how to create repositories, both master and work. Actually it is pretty simple and almost same with 11g which I told in this post.
So first impressions, when you download ODI 12c through this page, you will get odi_121200.jar (numbers can differ with time since it’s version number) and some opatches bundled with it. Actually it is a bit disturbing for me to have a jar file which is 1.8GB. I’d like to have an exe for Windows.
Anyway I had some problems with running this jar also, first I tried it on my VM which has 32bit Windows 7 and got an error that states it could not reach jar file. So I moved to my physical machine that is 64bit Windows 7 and OUI could not recognize the platform and exited everytime, until I download and install Java 1.7. So after solving the problem with Java, I moved to my VM again to solve other problem where it came out that my path is problematic, since my user name is Canburak Tümer, space created a problem to reach file. I created another user without space that can run the installer.
Finally I could see the installer UI. It was a pretty straight forward installation, I just selected enterprise and went on. After installation, I ran the ODI Studio, it has a really clean and elegant splash screen and it asks to migrate any user settings from old installations. After splash screen, ODI workbench has been load:
Creating Master Repository
As I mentioned before, repository creations are almost same with ODI 11g, we will start by clicking File > New and we will see screen below:
Select “Master Repository Creation Wizard” in ODI tab and click “OK”.
You will see screen above, where we will enter database information, schema where we will create repository and DBA user to run some of the creation scripts.
Define and confirm password for SUPERVISOR user. DO NOT FORGET THIS PASSWORD UNLESS YOU HAVE ANOTHER USER WITH SUPERVISOR PRIVILEGES. For this reason many ODI developers/admins make this password “SUNOPSIS” as an old habit. I prefer to have it as “SUPERVISOR” in my VM and personal development environment.
Select password storage as you wish. Then click finish, it will run scripts now to create master repository, it took around 4 minutes in my VM, probably it will take around 2-3 minutes in your physical machines. Now it’s time to create a connection to master repository.
Click on “Connect to Repository” then click to green plus in the pop-up window, then fill required information in the form. Use SUPERVISOR as ODI user and DB user which you have created the master repository with. Make sure you have selected “Master repository only” radio button. Then click “OK”.
ODI 12c will ask you to if you want to keep passwords in a secure wallet with a master password. I do not have enough information about this wallet yet, but I will learn and write another post about it. I prefer the less secure way which does not include the wallet. Now we have master repository and connection to master repository. Now it’s time to create work repository.
Creating Work Repository
To create work repository, connect your master repository then go to Topology tab and expand Repositories menu.
Right click to “Work Repositories” and click to “New Work Repository” from menu.
Insert connection information of schema which you want to create Work Repository in. (I had a problem with this step, actually I wanted to use odiw_c user but ODI 12c keeps filling the form in upper case so it gives invalid credentials error.)
In final step, insert repository name and select repository type.
You can also define a password for repository, which is different from ODI user password or DB user password. This password is just to secure the repository connection. When you click “Finish” it will run scripts to create work repository and will ask you if you want to create a connection to work repository. It will create a connection without ODI user information. So you will need to edit connection to insert ODI user information.
After all these steps we have installed ODI 12c and setup both master and work repositories for our environment. And we have a final view as below :
Now, it’s time to create our topology connections, models, projects; import or reverse engineer data sources; develop mappings (new name for interfaces), packages and more.
Welcome to ODI 12c, keep following my blog for further posts and please do not hesitate to contact me through comment form below.
So first post, first steps. We are going to create master and work repository for your Oracle Data Integrator. I assume that you have already have an ODI installation up and running and also an Oracle Database.
First I am going to create schemas for my repositories, you can put your repositories in one schema but it is common to have them in separate schemas. If you run the script below you will have two schemas for your repositories with all privileges.
create user ODIM_BLOG identified by ODIM_BLOG;
create user ODIW_BLOG identified by ODIW_BLOG;
grant all privileges to ODIM_BLOG;
grant all privileges to ODIW_BLOG;
Creating Master Repository
So if your schemas are ready, we can continue with the master repository. In the main ODI screen, go to File > New.
Now you’ll see three options, we’ll continue with Master Repository Creation Wizard, select it and click OK to continue.
Now it is time to setup our connection to database, you will need DBA connection to setup your repository. Fill in the required fields, you can see my blog environment’s information below. A little caution here: if you are going to use SYS user as DBA account, you have to fill DBA User area like ‘SYS as sysdba’ otherwise it will not connect. If you are using another DBA account then you don’t need to declare ‘as sysdba’. Hit the Test Connection button to see if you can connect.
There is another text box which asks for the repository ID, be sure to remember this ID, as you’ll also be asked for another ID while creating work repository. And it is important that all repositories have different IDs. As ODI does every operation with ID’s, it’s crucial not to have duplicated IDs.
Next step is the generating a password for SUPERVISOR user, this user will be our ODI admin until you will create another user with SUPERVISOR privileges. It is an old tradition to set this user’s password as SUNOPSIS, it is also a tribute like behavior for ODI’s creator Sunopsis. I set it as SUPERVISOR to remember it easily.
Here it goes, now we will wait until ODI finishes its job. If you check your master repository schema you can see it is being populated by SNP_% tables. SNP prefix is also a heritage from Sunopsis.
And here we have created our master repository. Now it is time to setup a connection and create a work repository.
Creating a Master Repository Connection
So for connecting master repository, click Connect to a Repository in ODI Home Screen. A little window will pop-up. This is the window where you can select between your predefined connection, edit them or create new connection. As we do not have any predefined connection, we will create one from scratch. Click on +(plus) sign to add a new connection.
Now this is the connection details window. First blank will be the alias for this connection so it should be something to remind you to which repository you are connecting. First login information is for ODI user we are going to use, which is the SUPERVISOR in our case. And the second login information is for the Master Repository’s database schema, this is ODIM_BLOG in our case.Make sure Master Repository only radio button is checked before testing the connection. Because we do not have any other repositories yet. Test your connection then connect your master repository with SUPERVISOR. It’s time to move a step further.
Creating Work Repository
When you are connected to aster repository go into Topology view and extend the repositories accordion. Right click on Work Repositories and click on the New Work Repository on pop-up menu.
Enter your schema information, test your connection and move on. In our case we are going to use ODIW_BLOG schema as we want our repositories to be separated.
Give a name and ID to your work repository. DO NOT forget to give a different ID from your master repository’s ID. Select your work repository type in this step.
There is two types of work repositories: Development and Execution. Execution repository does not have a development ability and does not have an active Designer tab. It only holds, scenarios, load plans, sessions and topology information. Development repository has an active Designer tab, which adds this repository to hold, create, edit, delete ability of interfaces, packages, variables, scenarios, knowledge modules, model objects like tables and views.
We will get into deeper about development and execution repository in another post. So I will make it short here.
Now let ODI to finish creating work repository.
Now ODI will ask if you want it to create a connection for new created work repository.
When you click Yes, it will ask for connection name, which we said it is connection alias. And when you click OK. It will create a connection.
Connecting to Work Repository
ODI has created a connection for us. When you disconnect your master repository. (Under ODI > Disconnect) You can connect to work repository, click on Connect to a Repository, select your work repository connection from dropdown list. Password field will be empty, enter your SUPERVISOR password and click connect.
As we do not have any other user yet, we will connect to work repository by SUPERVISOR user.
Thanks for your patience to finish reading. Next ODI post will be about creating a user in detail, and assigning profiles to a user.
If you have any questions, please use comment section to ask me.