Understanding IBM Tivoli Directory Integrator for IBM

Understanding IBM Tivoli Directory Integrator for IBM
Lotus Connections 3.0: A getting started guide
Chris Biega ([email protected]), Senior Certified Client Technical Professional, IBM Collaboration
Solutions, IBM
Charles Price ([email protected]), Advisory Software Engineer, IBM Collaboration
Solutions, IBM
August 2011
© Copyright International Business Machines Corporation 2011. All rights reserved.
Summary: The purpose of this white paper is to give users a good understanding of how IBM®
Lotus® Connections uses IBM Tivoli® Directory Integrator, so they can understand how to look at the
Lotus Connections AssemblyLines and understand what they do.
Table of Contents
1 Introduction.........................................................................................................................................2
2 Installation..........................................................................................................................................2
2.1 Tivoli Directory Integrator 7.0.......................................................................................................3
2.2 Install Fixpack 5........................................................................................................................10
3 Out-of-the-box Profiles population.....................................................................................................11
3.1 Pieces and parts........................................................................................................................11
3.2 Running the Installation wizard to populate Profiles..................................................................14
3.3 Running command files to populate Profiles.............................................................................25
4 Setting up the development environment..........................................................................................31
4.1 File system setup......................................................................................................................31
4.2 Extract TDI Solution directory....................................................................................................31
4.3 Configuration Editor...................................................................................................................32
5 Testing your configuration.................................................................................................................37
5.1 Verify properties are set correctly..............................................................................................37
5.2 Test your connections................................................................................................................37
5.3 Verify server setup.....................................................................................................................42
5.4 Run/test an AssemblyLine.........................................................................................................43
6 Logging/tracing.................................................................................................................................45
6.1 Troubleshooting.........................................................................................................................45
6.2 Logs..........................................................................................................................................47
6.3 Tracing......................................................................................................................................47
7 Connections 2.5 “Best Practice” approach........................................................................................48
7.1 Customizing AssemblyLines......................................................................................................49
1
7.2 Build the AssemblyLine.............................................................................................................51
7.3 Iterate through Master HR file...................................................................................................55
7.4 Look up additional LDAP information........................................................................................59
7.5 Look up additional database information...................................................................................62
7.6 Look up the manager UID.........................................................................................................66
7.7 Normalize names to remove accents (for search).....................................................................71
8 Connections 3.0 “Best Practice” approach.......................................................................................88
8.1 Profiles update philosophy........................................................................................................89
8.2 Design with upgrade in mind.....................................................................................................89
8.3 Project setup.............................................................................................................................92
8.4 Source repository connectors...................................................................................................94
9 Custom delete logic........................................................................................................................113
10 New Connectors...........................................................................................................................113
11 Samples files................................................................................................................................114
12 Conclusion....................................................................................................................................117
13 Resources....................................................................................................................................117
14 About the authors.........................................................................................................................117
1 Introduction
The purpose of this document is to give you a good understanding of how IBM® Lotus® Connections
uses IBM Tivoli® Directory Integrator (TDI). It does NOT teach you the TDI but rather provides enough
information so you can understand how to look at the Lotus Connections AssemblyLines and
understand what they do. You should always refer to the Tivoli Directory Integrator InfoCenter for a
more in-depth explanation.
What is Tivoli Directory Integrator? TDI is a software tool that lets you synchronize data across
multiple repositories. Within the Lotus Connections framework, you can use TDI to populate and/or
synchronize data from your LDAP directory, and possibly other sources, into the Profiles database.
You do this by running the Profiles population wizard or by manually populating the Profiles database
by running the supplied TDI AssemblyLines.
This paper walks you through installing TDI and the out-of-box methods for populating the Profiles
database. It then takes you through setting up and verifying your TDI development environment so
you can start making any customizations that may be necessary for your environment.
You'll learn about the AssemblyLines that are available to run and use as models for custom
AssemblyLines you may want to build. We'll end with best practices on how you should extend the
out-of-box AssemblyLines so you won't have to redo customizations when upgrades occur.
2 Installation
Let's begin by discussing the installation steps.
2.1 Tivoli Directory Integrator 7.0
We install TDI 7.0.5 on Microsoft® Windows® for this configuration, first installing TDI 7.0 and then
patching the install with Fixpack 5:
2
1.
On the installation media for TDI, execute the launchpad.exe file (see figure 1).
Figure 1. Launchpad.exe
2.
Select your language and click OK.
Figure 2. Select language
3.
Click the Tivoli Directory Integrator 7.0 Installer link (see figure 3).
3
Figure 3. Tivoli Directory Integrator Installation window
4.
The InstallShield Wizard Welcome window displays; click Next (see figure 4).
Figure 4. Welcome screen
5.
On the next window, click Next (see figure 5).
4
Figure 5. Search for previous TDI installations
6.
Accept the license agreement and click Next (see figure 6).
Figure 6. License agreement window
7.
Specify an Installation directory and click Next (see figure 7).
5
Figure 7. Specify installation directory
8.
Select Typical, and then click Next (see figure 8).
Figure 8. Select installation type
6
9.
Select the “Do not specify – use current working directory at startup time” option; click Next (see
figure 9).
Figure 9. Select to use current working directory
10.
Accept the default port values; click Next (see figure 10).
Figure 10. Port values to be used
11. Accept the default; click Next (see figure 11).
7
Figure 11. Register as a system service
12. Verify the information in the Summary window is correct; select Install (see figure 12).
Figure 12. Summary info window
13. The success window displays after the install is complete; uncheck the “Start the Configuration
Editor” options and click Finish (see figure 13).
8
Figure 13. Successful install message
2.2 Install Fixpack 5
Now that TDI 7.0 is installed, we need to patch it to Fixpack 5. To do this:
1.
Download FP5(7.0.0-TIV-TDI-FP0005.zip) from Fix Central and unzip it.
2.
Patch the update installer by taking the UpdateInstaller.jar file and replacing the copy in the
install_dir/maintenance directory. For example, replace the UpdateInstaller.jar in location
C:\IBM\TDI\V7.0\maintenance with the UpdateInstaller.jar in the zip file you just downloaded.
3.
Next, copy TDI-7.0-FP0005.zip to a temporary location. We copied ours to
C:\IBM\TDI\V7.0\fixpacks (see figure 14).
Figure 14. TDI-7.0-FP0005.zip copied to temporary location
9
Now let's install the fixpack. Currently the AMC Fix along with the LWI Update have not be added;
however, these products are rarely needed for use with Lotus Connections.
As of the writing of this paper, Connections 3.0 has been fully tested with and supports TDI 7.0 FP5.
With that said, it is generally recommended to keep current on ifixes. All samples in this paper were
tested on a TDI 7.0 FP6 system. For more details, refer to the IBM Support Downloads document #
7010509, “Recommended fixes for IBM Tivoli Directory Integrator (TDI).”
3 Out-of-the-box Profiles population
In this section we discuss how to populate Profiles with TDI.
3.1 Pieces and parts
It is important to understand all the “pieces and parts” of populating Profiles with TDI. In this section
we want to go over the most commonly changed files and explain what they control:
(Note that these files will be populated and/or used whether you are using the Population wizards or
manually running the population scripts.)
profiles_tdi.properties. This file contains the property values relevant to your configuration. See the
Connections Wiki for a full explanation of all parameters.
map_dbrepos_from_source.properties. Defines mappings from the enterprise directory [or input
source(s)] to the Profiles database. The mapping can consist of:
•
•
LDAP Attribute: Specifies the LDAP attribute to assign to the database field
JavaScriptTM function: Specifies a JavaScript function whose return value is the value assigned to
the database field. The function name is specified in between “{}” in the mapping file,
map_dbrepos_from_source.properties.
There are several default functions, but you can specify your own functions in
profiles_functions.js. The JavaScript functions take one argument that is the name of the
assigned field. For example, to normalize an email address to ensure it ends with acme.com,
assign this JavaScript function to PROF_MAIL:
function func_convertEmail(arg0){
var uid = work.getString("uid");
var email = uid+'@acme.com';
return email;
}
•
•
Null: Used to indicate that no data will be loaded into the field.
$variable; Variable will be acted on in the assemblyLine.
Table 1 lists the mappings between the profile field and column(s) in the corresponding database
tables.
10
Table 1. Mappings between the profile field and column(s)
Profiles application
Database Column
(EMPINST.EMPLOYEE unless stated)
Size
alternateLastname
PROF_ALTERNATE_LAST_NAME
(EMPLOYEE,SURNAME tables)
64
blogUrl
PROF_BLOG_URL
256
bldgId
PROF_BUILDING_IDENTIFIER
64
calendarUrl
PROF_CALENDAR_URL
256
courtesyTitle
PROF_COURTESY_TITLE
64
deptNumber
PROF_DEPARTMENT_NUMBER
24
description
PROF_DESCRIPTION
CLOB
displayName
PROF_DISPLAY_NAME
256
employeeNumber
PROF_EMPLOYEE_NUMBER
16
employeeTypeCode
PROF_EMPLOYEE_TYPE
256
experience
PROF_EXPERIENCE
CLOB
faxNumber
PROF_FAX_TELEPHONE_NUMBER
32
freeBusyUrl
PROF_FREEBUSY_URL
256
floor
PROF_FLOOR
16
groupwareEmail
PROF_GROUPWARE_EMAIL
128
guid
PROF_GUID
256
ipTelephoneNumber
PROF_IP_TELEPHONE_NUMBER
32
countryCode
PROF_ISO_COUNTRY_CODE
3
#isManager
PROF_IS_MANAGER
1
jobResp
PROF_JOB_RESPONSIBILITIES
128
email
PROF_MAIL
256
managerUid
PROF_MANAGER_UID
256
mobileNumber
PROF_MOBILE
32
nativeFirstName
PROF_NATIVE_FIRST_NAME
(EMPLOYEE,GIVEN_NAME tables)
256
nativeLastName
PROF_NATIVE_LAST_NAME
(EMPLOYEE,SURNAME tables)
256
11
orgId
PROF_ORGANIZATION_IDENTIFIER
64
pagerNumber
PROF_PAGER
32
pagerId
PROF_PAGER_ID
32
pagerServiceProvider
PROF_PAGER_SERVICE_PROVIDER
0
pagerType
PROF_PAGER_TYPE
16
officeName
PROF_PHYSICAL_DELIVERY_OFFICE
32
preferredFirstName
PROF_PREFERRED_FIRST_NAME
(EMPLOYEE,GIVEN_NAME tables)
32
preferredLanguage
PROF_PREFERRED_LANGUAGE
0
preferredLastName
PROF_PREFERRED_LAST_NAME
(EMPLOYEE,SURNAME tables)
64
secretaryUid
PROF_SECRETARY_UID
256
shift
PROF_SHIFT
4
distinguishedName
PROF_SOURCE_UID
256
telephoneNumber
PROF_TELEPHONE_NUMBER
32
timezone
PROF_TIMEZONE
64
title
PROF_TITLE
256
uid
PROF_UID
256
workLocationCode
PROF_WORK_LOCATION
32
surname
PROF_SURNAME
(EMPLOYEE,SURNAME tables)
128
givenName
PROF_GIVEN_NAME
(EMPLOYEE,GIVEN_NAME tables)
128
surnames
PROF_SURNAME
(EMPLOYEE,SURNAME tables)
128
givenNames
PROF_GIVEN_NAME
(EMPLOYEE,GIVEN_NAME tables)
128
loginId
PROF_LOGIN
256
logins
PROF_LOGIN
(EMPLOYEE,PROFILE_LOGIN tables)
256
map_dbrepos_to_source.properties. This defines mappings from the Profiles database to the
enterprise directory.
profiles_functions.js. This file contains all the functions used in the
map_dbrepos_from_source.properties file. Functions can be written to manipulate data to the correct
12
syntax. For example, if you want to combine two ldap attributes into one field, you'd write a function to
do that and then have map_dbrepos_from_source assign that function to the Profiles attribute.
collect_ldap_dns_generator.js. This file is used to specify custom filters for the collect_dns_iterate
and sync_all_dns_forLarge AssemblyLines.
validate_dbrepos_fields.properties. This file is used to specify up-front field validation. You can
have these invalidate a field instead of waiting for it to fail at the database layer.
3.2 Running the Installation wizard to populate Profiles
To do this:
1.
Copy the Wizard CD to the hard drive where you installed TDI (C:\downloads\Wizards) and run
the populationWizard.bat file (see figure 15).
Figure 15. populationWizard.bat file
2.
Click Next on the Profiles population wizard for Connections 3.0 Welcome window (see figure 16).
13
Figure 16. Profiles population wizard Welcome window
3.
Set the Tivoli Directory Integrator install directory (C:\IBM\TDI\V7.0); click Next (see figure 17).
14
Figure 17. Set TDI install directory location
4.
Select the Profiles database type used by Lotus Connections (IBM DB2® in our test environment)
and click Next (see figure 18).
15
Figure 18. Select Profiles database type
5.
Enter the details on how to connect to the database; click Next (see figure 19).
16
Figure 19. Profiles database properties window
6. Enter the details on how to connect to your LDAP directory; click Next (see figure 20).
17
Figure 20. LDAP server connection
7.
Specify the bind user name and password to query all users and attributes from your LDAP
directory; click Next (see figure 21).
18
Figure 21. LDAP authentication properties
8. Set the base entry for your LDAP directory and search filter to find all users; click Next (see figure
22).
19
Figure 22. Base distinguished names and filter for searches
9.
Check the mapping between the Profiles database and LDAP fields. A full list of LDAP attributes,
available variables, and functions will be in the drop-down list for you to use for your mappings
(see figure 23).
20
Figure 23. Profiles database mapping
Figure 24 below covers the population of the five supplemental tables that can be used during Profiles
population:
•
•
•
•
•
Countries. Value pair table that contains a country code / description pair.
Departments. Value pair table that contains a department code / description pair.
Organizations. Value pair table that contains an organization code / description pair.
Employee types. Value pair table that contains an employee type code / description pair.
Work locations. This is a multi value table containing a work location code / complete address
mapping.
See Section 3.3.1, “Manual Profile population,” below or the “Adding supplemental content to Profiles”
topic in the Connections Information Center for additional information.
This window also gives you the option to have Lotus Connections determine who is a manager, based
on whether you have a manager ID hooked to your record. If you do not map the isManager field, it is
recommended that you select Yes, to have Lotus Connections automatically mark your managers.
21
Figure 24. Optional database tasks
10. Review the summary details information, and click Configure to populate the Profiles database
(see figure 25).
22
Figure 25. Profiles population configuration summary
11. Once the population is complete, check for errors and click Finish (see figure 26).
23
Figure 26. Population completion summary
3.3 Running command files to populate Profiles
3.3.1 Manual Profile population
Before beginning, always download the latest Profiles TDI iFixes. All the source connector examples
at the end of this paper require a minimum of iFix L057683 (or most recent iFixes associated with IBM
Connections 3.0.1). When manually populating Profiles, at a minimum you must do the following:
1.
First, update the profiles_tdi.properties file in the TDISOL\TDI directory. The fields outlined in red
must be adapted for your environment (see figure 27). Passwords will be encrypted after the first
time this is run.
24
Figure 27. profiles_tdi.properties
2.
Next, you must update map_dbrepos_from_source.properties. This file is used to map the LDAP
attributes to the employee table in the Profiles database.
3.
You can now populate the Profiles database. By this time you must have the artifacts specified
above copied to the TDISOL\TDI directory. Run these three commands to completely populate
the database (see figure 28):
•
•
•
collect_dns. Creates the collect.dns file with all the users in the LDAP.
populate_from_dn_file.bat. Populates the main Profiles tables (EMPLOYEE, SURNAME,
GIVEN_NAME).
mark_managers. Marks all managers with a ‘Y’ so you will receive the People Managed link if
the user is a manager.
Figure 28. Commands in Command Prompt window
25
4.
You can run six additional commands to populate the supplemental user data tables for more
complete information. Look at the data in the .csv files; if you do not like the data (or want to
customize for a particular industry), now is the time to change it.
•
Supplemental tables (optional):
o
o
o
o
o
o
fill_country. populates the COUNTRY table from the isocc.csv file
fill_department. populates the DEPARTMENT table from deptinfo.csv file
fill_emp_type. populates the EMP_TYPE table from the emptype.csv file
fill_organization. populates the ORGANIZATION table from the orginfo.csv file
fill_workloc. Populates the WORKLOC table from the workloc.csv file
load_photos_from_file. Mass-loads photos for all users
You now have a fully loaded Profiles database
3.3.2 Common AssemblyLines
Below is a list of common AssemblyLines that are used for the population and maintenance of
Profiles. For a complete list, see “Batch files for process Profiles data”.
•
collect_dns. This executes the collect_ldap_dns AssemblyLine. This AssemblyLine works off the
source_ldap_xxxxx ldap properties specified in the profiles_tdi.properties file. The capacity of the
standard collect_dns AssemblyLine provided with Lotus Connections is 100,000 users.
In some cases, you can modify the maximum number of entries returned from the LDAP or adjust
the source_ldap_page_size parameter in the profiles_tdi.properties file (must be supported by the
LDAP). If this does not work, you can use a special AssemblyLine (collect_dns_iterate) to
populate your Profiles database from your LDAP directory.
•
collect_dns_iterate. This executes the collect_ldap_dns_iterate AssemblyLine (see figure 29).
Instead of using the search base specified in the profiles_tdi.properties files, it uses the
collect_ldap_dns_generator.js file to retrieve search criteria for a batch of records.
If you look at the sample collect_ldap_dns_generator.js, you will see it goes a character at a time,
returning names that begin with the two characters listed. You would modify this to make sense
for your list of users. Note that the total returned in each chunk must be small enough to conform
to the size parameters as would collect_dns.
Figure 29. collect_ldap_dns_iterate
26
•
populate_from_dn_file. This executes the populate_from_dns_file AssemblyLine. Keep in mind
that this AssemblyLine is accumulative, meaning that if the record is there, it will update it; if it
does not exist, it will add it. This assembly does NO deletes.
This AssemblyLine determines whether a record is new by the uid. If the uid changes on a record
(which may happen for a name change, etc.), you must use the sync_all_dns AssemblyLine to
make the update. In sync_all_dns, you can specify what attribute (sync_update_hash_field in the
profiles_tdi.properties) to use for determining whether a user should be updated or added.
•
mark_managers. This executes the mark_managers AssemblyLine. It goes through the
employee table and determines whether you are a manager.
•
load_photos_from_files. This executes the load_photos_from_files AssemblyLine. It relies on a
default input file called collect_photos.in (can be overridden in the profiles_tdi.properties file) to
point to the location of the photos. It reads the uid/photo location from the input file and then calls
the PhotoConnector to do the insert. Users must already exist in the Employee table for the photo
to be added. Its format is shown in figure 30.
Figure 30. Format of load_photos_from_files
load_pronounce_from_files. This executes the load_pronounce_from_files AssemblyLine. It relies
on a default input file called collect_pronounce.in (can be overridden in the profiles_tdi.properties file)
to point to the location of the photos. It reads the uid/pronunciation location from the input file and then
calls the PronunciationConnector to do the insert. Users must already exist in the Employee table for
the pronunciation to be added. Its format is similar to the photos input file above.
fill_country. This executes the populate_country AssemblyLine and relies on a default input file called
isocc.csv. Its format is shown in figure 31.
27
Figure 31. Format of fill_country
fill_department. This executes the populate_department AssemblyLine and relies on a default input
file called deptinfo.csv. Its format is shown in figure 32.
Figure 32. Format of fill_department
fill_emp_type. This executes the populate_emp_type AssemblyLine and relies on a default input file
called emptype.csv. Its format is shown in figure 33.
28
Figure 33. Format of fill_emp_type
fill_organization. This executes the populate_organization AssemblyLine and relies on a default input
file called orginfo.csv. Its format is shown in figure 34.
Figure 34. Format of fill_organization
fill_workloc. This executes the populate_workloc AssemblyLine and relies on a default input file
called workloc.csv. Note that every attribute does NOT need to be included. Its format is shown in
figure 35.
Figure 35. Format of fill_workloc
delete_or_inactivate_employees. This executes the delete_employee_records AssemblyLine. It
relies on a default input file called delete_or_inactivate_employees.in (can be overridden in the
profiles_tdi.properties file). This AssemblyLine also relies on the sync_delete_or_inactivate property
being set in the profiles_tdi.properties file. Its format is shown in figure 36.
Figure 36. Format for delete_or_inactivate_employees
dump_photos_to_files. This executes the dump_photos_to_files AssemblyLine and gives you an
easy way to dump all employee photos from the database to a file.
dump_pronounce_to_files. This executes the dump_pronounce_to_files AssemblyLine. This gives
you an easy way to dump all employee pronunciation from the database to a file.
process_draft_updates. This executes the process_draft_updates AssemblyLine and synchronizes
changes from the Profiles database back to the LDAP directory.
29
sync_all_dns. This executes the sync_all_dns AssemblyLine and updates the Profiles database to
capture changes to the LDAP directory. The synchronization process includes updates to employee
records and additions/deletions of records.
sync_all_dns_forLarge. This executes the sync_all_dns_forLarge AssemblyLine and synchronizes
changes from the LDAP directory back to the Profiles database. Use this batch file when the LDAP
directory has a data size limitation; some LDAP directories reject requests to collect large amounts of
data in one operation, such as fetching 100,000 records at a time.
Examine your LDAP server settings to discover the largest number of records that you can collect. If
the figure is smaller than the number of records in the LDAP directory, you must use this batch file.
4 Setting up the development environment
Next we set up our development environment for Assembly Development.
4.1 File system setup
Set up your file structure as specified in the “Setting up your development environment” topic in the
Connections InfoCenter. For example, we set up the development structure shown in figure 37.
Figure 37. Development structure
4.2 Extract TDI Solution directory
Next we need to extract the tdisol.zip file to C:\IBM\TDIProject\V7GA-20100927\TDISOL, after which
we will import this into a New Project. This can be the same TDISOL used in the wizards or the
manual process discussed above.
30
4.3 Configuration Editor
Start the Configuration Editor (ibmditk -s c:\ibm\TDIProject\V3GA-20100927\TDISOL\TDI) in a
Command Prompt window (see figure 38). It is important that you start it via the command line and
specify the solution directory at startup.
NOTE: If you currently run the tdienv script manually from the Solution directory and then execute
<tdi home dir>ibmditk, that still will work as it did in previous releases.)
Figure 38. Start Configuration Editor
It will prompt you for a workspace. Point to the workspace that we just created above and then click
OK (see figure 39).
Figure 39. Select a workspace
Now, we must create a new Project:
1.
Once TDI opens, select File – New– Project (see figure 40).
31
Figure 40. Name the new project
2.
On the Select a wizard window, open IBM Tivoli Directory Integrator and select Project (see figure
41); click Next.
Figure 41. Select a wizard
3.
Give the new project a name, and click Finish (see figure 42).
32
Figure 42. Name the new project
4.
Import the Connections Configuration file (profiles_tdi.xml) from where we just
unzipped it above. Note that you may need to close the Tivoli Directory Integrator Welcome
window (see figure 43).
Figure 43. TDI Welcome window
5.
Select the project that we just created, right-click, and then select Import (see figure 44).
33
Figure 44. Import project
6.
Select Configuration under IBM Tivoli Directory Integrator; click Next (see figure 45).
Figure 45. Select Configuration
7.
Fill in the Configuration File and then click Finish. Your Connections Configuration is now
imported into your project (see figure 46).
34
Figure 46. Select objects to import
You now have the default AssemblyLines in the Configuration Editor (see figure 47).
Figure 47. Default AssemblyLines
35
5 Testing your configuration
Now we must test the configuration.
5.1 Verify properties are set correctly
Let's first look at the properties file (see figure 48). When you started the Configuration Editor, you
started with the -s option that points to the Solution directory containing your changed
profiles_tdi.properties file. You should see your changed properties here; if not, make sure you used
the -s option when starting the Configuration Editor (ibmditk -s c:\ibm\TDIProject\V3GA20100927\TDISOL\TDI).
Figure 48. Properties window
5.2 Test your connections
Next we verify that we can connect to an LDAP. In the Connections30GA project that we just created
and into which we imported the Configuration file, do the following:
1.
Expand the assemblyLines, select collect_ldap_dns, select the ldap_iterate connector, and test
your connection (see figure 49).
2.
Import the Connections Configuration file (profiles_tdi.xml) from where we just unzipped it above.
3.
Test the LDAP connection:
a. Expand AssemblyLines
b. Select collect_ldap_dns
36
c. Select the ldap_iterate connector
d. Select Input Map
e. Click on the Connect button.
Figure 49. collect_ldap_dns window
You should now see a window similar to that shown in figure 50, with the attribute names that
were brought back from the connect.
37
Figure 50. Attribute names
f. Click the Next button; at this time you are actually reading the first LDAP entry. You will see
the sample values returned (see figure 51).
Figure 51. Sample values
g. Click the Close button to close the connection
Now let's test the DB2 connection (see figure 52):
38
a.
b.
c.
d.
Expand AssemblyLines
Select dump_employee_records
Select the load connector
Select Input Map
Figure 52. dump_employee_records window
e. Click the Connect button. At this time you should see a window similar to that shown in
figure 53, in which the attribute names that were brought back from the connect are shown.
39
Figure 53. Attribute names
f. Click the Next button. At this time you are actually reading the first table entry. You will see
the values returned (see figure 54).
Figure 54. Sample values
g. Click the Close button to close the connection.
You now have successfully tested your Configuration setup. You can do the same type of test for any
connector, including filesystem file read, IBM Lotus Domino®, etc.
40
5.3 Verify server setup
During install, the server occasionally has an issue setting up the port correctly. If the Default server
does not start and you get messages like that shown below, then you most likely have an issue with
the API naming port:
Wed Sep 29 16:02:15 CDT 2010 CTGDIC076I
Wed Sep 29 16:02:34 CDT 2010 CTGDIS210W
initialization: {0}.
Wed Sep 29 16:02:34 CDT 2010 CTGDKD002E
'api.remote.naming.port': 'null'.
Wed Sep 29 16:02:34 CDT 2010 CTGDIC077I
Local development server launched ...
An error has occurred on Server API
Invalid value specified for
Process exit code: '0'
To correct this, you need to change/verify the port setting in multiple places. First, verify that the port in
the Server document is correct. In the example shown in figure 55, you can see that it is set at null.
Figure 55. Server document
We need to change that 1099 value (see figure 56) and then save the Server document.
41
Figure 56.
The Server itself gets the port to use from its properties file. If you have a Solution directory, then this
will be solution.properties in that folder. If you are using the install folder as the Solution directory, then
it will be in %TDI_INSTALLDIR%/etc/global.properties.
For our example we used a Solution directory called C:\IBM\TDIProject\V3GA-20100927\TDISOL\TDI,
so look in the solution.properties file there. The file contains a number of settings. The property you
are looking for is “api.remote.naming.port=1099” (see figure 57).
Figure 57. The “api.remote.naming.port=1099” property
Once both these changes are made, the Default server should start.
5.4 Run/test an AssemblyLine
Once again, we use the collect_ldap_dns AssemblyLine to show how you might run/test an
AssemblyLine from the Configuration Editor. First, make sure your server is started and
collect_ldap_dns is selected. Then start a debug session to allow you to step through your
AssemblyLine (see figure 58).
42
Figure 58. Start debug session
TIP: All connectors have a Detailed Log option on the Connection tab (see figure 59). Enable this
(check mark) to get additional information on what data is being returned.
Figure 59. Detailed Log option
43
6 Logging/tracing
6.1 Troubleshooting
The Profiles TDI solution that's used to perform actions on your Profiles deployment is composed of
several interconnected components. When you are attempting to determine the cause of an error, it is
sometimes necessary to first determine which layer is most likely to be involved in the error and how
to enable tracing specific to this component to resolve the problem.
This architecture has changed in Lotus Connections 3.0 compared to both 2.5 and 2.0.x. In
Connections 2.0.x all Profiles-related logic was contained in the config. Connections 2.5 moved much
of the business logic to centralized jars, and Connections 3.0 included the addition of connectors and
adapters, making the solution more easily extended and customizable by users (see figure 60).
Figure 60. Connections 3.0 TDI architecture
Profiles-specific output can come from the TDI config itself (profiles_tdi.xml) or from the component
jars that are part of the solution. These jars contain business layer functionality that is shared by the
Profiles Web application as well as the Profiles TDI solution.
Adjusting trace levels for these backend components is quite similar to the process used for the Web
application, and the same class hierarchical subsets can be used to view various tracing level outputs.
6.1.1 Error codes
Error messages from all Profiles components, whether TDI related or not, use the same error code, |
CLFRN. All error codes with this prefix are documented in the Lotus Connections Information Center.
In the log files you will see errors from other components, and these can be found in the
corresponding product documentation. For example, TDI error codes begin with the prefix CTGDIS.
Besides the component prefix of the error code, the last letter designates the severity. A code ending
with an I, for example, is an informational message, while Error and Warning codes are designated by
E and W, respectively. When you reference the Info Center documentation concerning a code,
additional information is provided explaining any action needed.
6.1.2 Trace messages
Tracing messages, on the other hand, are neither published nor translated output. A user would
enable tracing when instructed by Lotus Support or when troubleshooting a problem independently.
This document gives an overview of methodology to enable trace logging when troubleshooting TDI
configuration and functional problems.
44
6.1.3 Log files
The log files produced by running Profiles TDI AssemblyLines are all stored in the logs subdirectory of
the solution. Most AssemblyLines create a log file specific to that AssemblyLine, for example,
PopulateDBFromDNFile.log or SyncUpdates.log. When a task is run a subsequent time, based on the
logging implementation of the AssemblyLine, the logs either are renamed with an incremented trailing
digit or named with a date suffix to preserve past logs.
All other output, as well as some AssemblyLine output, is recorded in the logs\ibmdi.log file. Ibmdi.log
is where "standard output" and "standard error" are recorded by default. This can be modified by
configuration file etc\log4j.properties. For further information, see the TDI Info Center.
6.1.4 Enabling tracing in TDI
Trace settings are configured in the etc\log4j.properties file. More information can be found on these
configurations in the TDI Info Center.
Tracing is enabled via the log4j java logging component. There are different tracing levels and, if a
given level is enabled, all levels lower in severity are also enabled. The different levels in order of
lowest to highest severity are ALL > TRACE > DEBUG > INFO > WARN > ERROR > FATAL > OFF.
Profiles service layer logging
If you are troubleshooting a Profiles or TDI problem and are unsure in what area the problem may lie,
the best strategy is to start with a general trace setting at the lowest severity level. If a targeted task is
run with this setting, then more information can be gained on where the problem may lie that would
help narrow down the trace configuration.
The most general trace setting to use to encompass the set of Profiles service classes is
log4j.logger.com.ibm.lconn.profiles=ALL
To confine tracing to just the TDI operations, set the following line in the etc\log4j.properties file:
log4j.logger.com.ibm.lconn.profiles.api.tdi=ALL
The classes in this place in the hierarchy are primarily used as the interface between the
AssemblyLines and connectors and the service layer. This level of tracing might also identify issues in
the calling TDI constructs.
Event log tracing is also a worthwhile area to trace. Whenever database operations are performed,
they are also recorded in the event log. By enabling tracing at this level, you can see indirectly what
was done via the service-level API to the database.
To enable event log tracing, set the following line in the etc\log4j.properties file:
log4j.logger.com.ibm.lconn.profiles.internal.service=ALL
This setting is quite verbose, so be sure to enable it for targeted operations.
Database layer logging
Although the database layer is not part of the Profiles service layer, it is often quite useful to enable
tracing at this layer to troubleshoot problems. To enable database tracing, set the following line in the
etc\log4j.properties file:
45
log4j.logger.java.sql=ALL
This setting is also quite verbose, so enable it for targeted operations.
6.1.5 Resources
The Lotus Connections Info Center contains TDI troubleshooting information in the Troubleshooting
section under the "Troubleshooting problems with the Tivoli Directory Integrator" topic.
The complete set of error codes is also found in the Troubleshooting section. The Profiles TDI error
codes share the same prefix as the rest of the Profiles components and are found on the "Profiles
error messages" page.
6.2 Logs
Every time you run an AssemblyLine, entries are written to log files, which are written to the
<tdisol>\logs directory. Log files of interest are as follows:
ibmdi.log. This log file contains the information for the latest run AssemblyLine
PopulateDBFromDNFile.log. Contains information about the latest populate_from_dn_file
AssemblyLine.
LoadPhotoFromFiles.log. Contains information about the latest load_photos_from_file
AssemblyLine.
LoadPronounceFromFiles.log. Contains information about the latest load_pronounce_from_file
AssemblyLine.
DeleteEmployeeRecords.log. Contains information about employees that were deleted or
inactivated.
Running sync_all_dns and sync_all_dns_forLarge produces the following set of log files:
•
•
•
•
•
•
•
Hash_LDAP_Iterate.log
HashDB.log
sync_all_dns_it.log
syncAllDNSProcessDelete.log
SyncUpdates.log
ibmdi.log
PopulateDBFromDNFile.log
When debugging issues, always start with looking at the ibmdi.log.
6.3 Tracing
Before we can start discussing testing our AssemblyLines, it is important to understand how data is
stored and transported within the system.
6.3.1 Entry object
One of the cornerstones of understanding Tivoli Directory Integrator is knowing how data is stored and
transported within the system. This is done by use of an object called an entry, and this entry object
can be thought of as a "JavaTM bucket" that can hold any number of Attributes: none, one, or many.
46
6.3.2 Entry types
There are a number of data objects that reside in AssemblyLines that follow the Entry data model:
Work. This is the aforementioned Entry that travels from component to component in the
AssemblyLine, carrying data between them. Its pre-registered variable name is work, and it's available
for use in scripting almost everywhere.
Conn. This is the Entry-like object that a Connector uses as an intermediary between the connected
system and the AssemblyLine, before you make the data, or a subset thereof, available in the Work
Entry. The process of moving data between Conn and Work is called Attribute Mapping.
Its pre-registered variable name is “conn,” and it's available for use in scripting inside many of the
Hooks in Connectors and Function components.
Current. This Entry-like object is available inside certain Hooks in Connectors in Update mode. It
holds the data from the connected system before any updates have been applied to it. Its preregistered variable name is “current.”
Error. This Entry-like object exists only in certain Hooks in Components, when some error condition
has occurred, and the relevant Error Hook has been invoked. It contains information about the actual
exception that was thrown, with possibly some additional variables and data, enabling you to pinpoint
what exactly caused the error. Its pre-registered variable name is “error.”
You can dump the Entry object details to the ibmdi.log file, to make it easier to debug your
AssemblyLines, and you can add logmsg to any of your hooks or script components:
task.logmsg("****** Error *******");
task.dumpEntry(error);
task.logmsg("****** Work *******");
task.dumpEntry(work);
If you find an error and want to go to the next iteration, you may or may not want to do this:
system.skipEntry();
try {
task.logmsg("****** CONN *******");
task.dumpEntry( conn )
} catch (e) { // Do nothing }
try {
task.logmsg("****** CURRENT *******");
task.dumpEntry( current )
} catch (e) { // Do nothing }
7 Connections 2.5 “Best Practice” approach
This section demonstrates a simplistic yet realistic example of a custom AssemblyLine, in which a
user populates Profiles from three different datasources. LDAP contains much of the user information,
but we also need to import information from an export of a Human Resources (HR) system as well as
a relational database table.
47
For simplicity's sake, we only discuss a one-way pull to update the Profiles database. Variations of
everything in the example have been seen at multiple customer sites.
7.1 Customizing AssemblyLines
When building a custom AssemblyLine, the first thing to consider is “where is my data coming from?”.
Is it from the LDAP, an HR export file, a database, etc.? In most instances it is coming from a
combination of datasources. For this example our main input file is an export from an HR system,
which is generally a comma-delimited file (CSV). We then look up additional information from an
LDAP and a relational table.
Next you must determine what information you want to be able to see on a person's profile. The best
way to do this is to get a sample of the data in the different sources. So, for our example you would
get a sample of the HR .csv file, a representative LDAP Import File (LDIF) record, and a sample
database row. From these you can see what information you want to load into Profiles and determine
if any of the data needs to be manipulated.
After you have this information, you are now ready to design and code your custom AssemblyLine.
Let's now look at our different input files and do a quick design of the AssemblyLine.
Sample input files data are shown in figures 61—63.
Figure 61. HR input file (employeeData.in)
48
Figure 62. LDAP import file
Figure 63. Relational HR data (location table)
In our example, the master data is the HR file. Only users in the HR file should be loaded into Profiles,
so we must iterate through the HR file and then look up additional information in the LDAP and
database files.
Both the CSV and LDIF have manager information, but the CSV is the master, and the LDIF may not
be accurate. Since the CSV has the manager employee ID versus the UID, we must do a lookup in
the LDAP to find the manager's UID.
We also notice a couple of things about the data; specifically, both first name (givenname) and
surname (sn) can have accented or double-byte characters. We need to be able to look up these
users both with and without the accent.
So, on a high level, we must do the following (see figure 64):
1.
2.
3.
4.
5.
Copy assemblyLine and define necessary resources
Iterate through Master HR file
Look up additional LDAP information
Look up additional database information
Look up the manager UID
49
6.
7.
8.
Normalize names to remove accents (for search)
Map all fields
Update the Profiles database
Figure 64. High-level approach
7.2 Build the AssemblyLine
Now let's actually build the AssemblyLine. First, we determine whether there is a similar AssemblyLine
that we can copy. If we look at the populate_from_dns_file, it has a lot of what we need, so we copy
this AssemblyLine and start with it:
1.
Clone the populate_from_dns_file so we can use it as a starting point:
a) Right-click on the populate_from_dns_file assemblyLine, and select Copy (see figure 65).
50
Figure 65. Copy the populate_from_dns_file
d
b) Right-click again in the side Navigator and click Paste. Enter a new name for the copied
AssemblyLine (see figure 66).
Figure 66. Enter new name
c) Look at the AssemblyLine we just created by double-clicking the new AssemblyLine in the
Navigator side bar. We want to copy this one because of the code in the hooks; it does a lot of
51
the initialization that we want to reuse. Look through the different hooks and take note of the
initialization code and error handling (see figure 67).
Figure 67. AssemblyLine hooks
2.
Create a properties file to contain the relevant source connection information:
a) Create a file in the <tdisol> directory called IBMPoCprofiles_tdi.properties. Your passwords
are encrypted the first time you enter them, but if you don't want them encrypted you can
remove the {encr} and type the passwords in clear text (see figure 68).
Figure 68. IBMPoCProfiles_tdi.properties file
b) Next, define that properties file to your current TDI project by selecting New Property Store
and naming it IBMPoC (see figure 69).
52
Figure 69. New Property Store
c) Select the Connector tab and, under Configuration – Properties Connector, click Select and
browse to the IBMPoCprofiles_tdi.properties file (see figure 70).
d) Click the Download button to load the properties file information from the server.
53
Figure 70. Load properties file information
7.3 Iterate through Master HR file
Now let's look at the Editor tab to see the values in the properties file (see figure 71). We can use this
file with our multiple source connectors.
54
Figure 71. Editor tab
1.
Close this new properties file, saving the changes.
2.
In the Navigator window, open our project (Connections30GA) – AssemblyLines, and double-click
on the copied AssemblyLine (IBMPoC_populate_from_csv_file).
3.
Rename iterate_over_dns to iterate_through_HR_file, and change the parser type to a CSV
parser that can read our input HR file. Our CSV has a comma for a delimiter, and we want to
ignore the headers.
4.
Follow all the steps shown in figure 72; your window should look similar to that shown in this
figure.
55
Figure 72. Parser tab steps
5.
Switch to the Input Map tab, delete ALL existing mappings, and then add a mapping to all
attributes (see figure 73).
56
Figure 73. Input Map tab
6.
On the Connection tab, map to IBMPoCprofiles_tdi.property file values (see figure 74). Click File
Path (property), to bring up the Expression Editor.
Figure 74. Connection tab
57
7.
Select the Use Property option, and then select “hr_input_file” from the IBMPoC file (see figure
75). After this, you can test your connector the same way we tested the connectors above.
Figure 75. File Path window
7.4 Look up additional LDAP information
Now let's work on the LDAP lookup. The changes here are minor:
1.
Under Data Flow, select lookup_dn, right-click it, and rename lookup_dn to
lookup_ldap_user_info (see figure 76).
58
Figure 76. Rename lookup_dn
2.
On the Link Criteria tab, change the lookup parameter to uid equals $UserID (just manually type
in $UserID), as shown in figure 77.
Figure 77. Link Criteria tab
59
3.
On the Connection tab, use the Expression Editor to point to the IBMPoC properties file (see
figure 78).
Figure 78. Connection tab
4.
Test your connector to make sure you can read from your LDAP, just like you tested your ldap/db2
connectors above. Go to the Input Map tab and select Connect, then Next, then Close. Make sure
data is returned.
7.5 Look up additional database information
Create an HR location connector (DB2 connector):
1.
Add a JDBC lookup connector by right-clicking on Data Flow and selecting Add Component (see
figure 79).
60
Figure 79. Add Component
2.
Select the JDBC Connector component, set the Name field as “lookup_relational_user_info”, and
set the Mode field as “Lookup”. Click Finish (see figure 80).
61
Figure 80. Choose Component window
3.
Drag the connector directly under lookup_ldap_user_info (see figure 81).
Figure 81. Drag connector under lookup_ldap_user_info
4.
Enter the HR database Connection information, as shown in figure 82.
62
Figure 82. Connection information
5.
Set the Link Criteria information as shown in figure 83.
Figure 83. Link Criteria info
6.
Set the Input Map information as shown in figure 84.
63
Figure 84. Input Map info
7.
Test your connector to make sure you can read from your LDAP, just like you tested your ldap/db2
connectors above: On the the Input Map tab, select Connect, then Next, then Close, and make
sure data is returned.
7.6 Look up the manager UID
Now we add the lookup manager connector, looking up the manager based on a passed-in manager
employee ID:
1.
Add an LDAP lookup connector by right-clicking on Data Flow and selecting Add Component (see
figure 85).
Figure 85.
2.
Select the LDAP Connector component, set the Name as “lookup_manager_uid” and set
the Mode as “Lookup”. Click Finish (see figure 86).
64
Figure 86. Choose Component window
3.
Drag the connector under the lookup_relational_user_info connector (see figure 87).
Figure 87. Drag connector under lookup_ldap_user_info
4.
Set the Connection attributes as shown in figure 88.
65
Figure 88. Set Connection attributes
5.
Set Link Criteria to employeeNumber equals $MANAGERID, by just typing it in (see figure 89).
Figure 89. Set Link Criteria
6.
Set Input Map as shown in figure 90.
66
Figure 90. Set Input Map
Now let's test what we have:
1.
Click the test button, to start walking through the assemblyLine (see figure 91). You are now in
debug mode.
Figure 91. Test button
2.
First you want to confirm that you have “work” in your watch list (see figure 92). If it is not there,
you can add it by selecting it from the “Edit Watch List” selector. Next, step over the connectors till
you get to callSyncDB_mod.
67
Figure 92. Watch List
3.
Look at the information in the work object (see figure 93). This is the information that is being
passed to callSyncDB_mod.
68
Figure 93. Work object info
7.7 Normalize names to remove accents (for search)
Let's add one more change to our AssemblyLine. Occasionally it is necessary to have accented
characters in the name, and you want be be able to search for the name with and without the
accented characters.
Profiles has two tables (SURNAME and GIVEN_NAME) in which names are stored for quick lookup
while you're in Profiles. We'll add code to normalize the name, if it has an accent, and add both the
accented and unaccented name. The first thing we must do is add a couple of functions that we call to
do the work:
1.
In the Navigator sidebar, expand Resources, right-click on Scripts, and select New Script (see
figure 94).
69
Figure 94. Select “New Script”
2.
Give the script a name (IBMPoC_normalizeName), as shown in figure 95. Click Finish.
Figure 95. Specify Name
70
3.
Next, copy in the two required functions shown in listing 1 into the Script Editor tab (see figure
96).
Listing 1. Required functions
/* In general the normalization routines take an accented character, for
example, ü(00fc)
/* and normalizes it to 0075 0308(small u 0075 diaeresis 0308),
diaeresis is removed. */
function isAccented(strValue) {
if(strValue) {
var newValue = stripAccentedChars(strValue);
if(strValue.equals(newValue)) {
return false;
}
else {
return true;
}
}
return false;
}
function stripAccentedChars(strValue) {
if(strValue) {
return com.ibm.icu.text.Normalizer.normalize(
strValue,
com.ibm.icu.text.Normalizer.NFD)
.replaceAll("\\p{InCombiningDiacriticalMarks}+", "");
}
return strValue;
}
Figure 96. Script Editor tab
71
then the
4.
Select the Configuration tab and enable the Implicitly Included option (see figure 99). Save and
close the new script.
Figure 99. Configuration tab
Now open the IBMPoC_populate_from_csv_file AssemblyLine and create an Attribute loop, to loop
through the surname name attribute:
1.
Right-click on Data Flow and select Add Component (see figure 100).
72
Figure 100. Select Add Component
2.
Filter the components by “Control/Flow”, select the Attribute Value Loop, and give it a name; click
Finish (see figure 101).
73
Figure 101. Select Type Filter
3.
Select Yes, to add the IF branch (see figure 102).
Figure 102. Branch Conditions window
4.
Filter the components by “Control/Flow”, select the IF component, and give it a name; click Finish
(see figure 103).
74
Figure 103. Select Type Filter
5.
Now we specify the attribute we will loop through. By default the sn attribute is a multi-value
attribute in ldap; it can contain from 1 to many values. We will loop through sn and define a
variable called snValue to be our loop variable (see figure 104).
75
Figure 104. Attribute Value Loop
6.
Next we define the IF clause. For this example we write a script, selecting the Script button and
adding the code shown in figure 105. This passes the current snValue value into the isAccented
function, which returns true if the name contains accents, false otherwise.
76
Figure 105. Script code
7.
Next we need to add the logic to normalize the name (see figure 106); for example, let's take the
name “fòrd” and normalize it to “ford”.
77
Figure 106. Add if logic
8.
Filter the components by “Scripts”, select the Empty Script component, and give it a name; click
Finish (see figure 107).
78
Figure 107. Select Type Filter
9.
Add the code shown in listing 2 as shown in figure 108.
Listing 2. add_to_snAttribute code
// get sn attribute
var anAttr = work.getAttribute("sn");
// add normalized name
anAttr.addValue(system.trim(stripAccentedChars(work.getString("snValue"))).toLower
Case());
79
Figure 108. Add code
10. Click Close. All the code to normalize surnames is now complete.
Create an Attribute loop, to loop through the givenName name attribute:
1.
To define Attribute Loop, right-click on Data Flow, and select Add Component; click Finish (see
figure 109).
80
Figure 109. Attribute Value Loop
2.
Select Yes, to add the IF branch (see figure 110).
Figure 110. Branch Conditions window
3.
Filter the components by “Control/Flow” and then select the IF component. Give it a name and
click Finish (see figure 111).
81
Figure 111. Select IF component
4.
Give it the attribute to loop through; in our case, givenName, and then a loop variable called
gnValue (see figure 112).
82
Figure 112. gnValue loop variable
5.
Add the script logic by selecting the Script button and then add the logic shown in figure 113.
83
Figure 113. Add script logic
6.
Add the logic for the IF statement by right-clicking on IF:if_gname_accented and choosing Add
Component – Scripts – Empty Script. Give the script a name (add_to_gnAttribute) and click
Finish. Now you can add the script in listing 3 and shown in figure 114.
Listing 3. add_to_gnAttribute script
// get givenName attribute
var anAttr = work.getAttribute("givenname");
// add normalized name to givenName attribute
anAttr.addValue(system.trim(stripAccentedChars(work.getString
("gnValue"))).toLowerCase());
84
Figure 114. Add script
7.
Move the two FOR-EACH loops listed above callSyncDB_mod, as shown in figure 115.
85
Figure 115. Move scripts
Now you can test your final AssemblyLine. To do this you may want to change one of your user
names, giving it an additional sn or givenname attribute. Let's add an accented name. For example,
suppose you have a user name “ford”. Go to their LDAP record and add the following:
Original
sn = ford
Add another sn attribute called
sn = fòrd2
Now we can run through the AssemblyLine in debug mode to ensure everything works:
1.
Click the Start Debug session button (see figure 116).
86
Figure 116. Start Debug session
2.
Step over Iterate_through_HR_file and lookup_ldap_user_info at this point. If you open work and
sn, you should see two records, ford and fòrd2, as shown in figure 117.
Figure 117. ford and fòrd2 records
3.
Step over lookup_relational_user_info and lookup_manager_uid.
87
4.
Step over add_to_snAttribute; you should now see three attributes for this user (see figure 118):
Ford
fòrd2
and ford2 that was added by our code.
Figure 118. Ford, fòrd2, and ford2 attributes
4.
You can either continue stepping through the assemblyLine or let it run.
You should now have a good understanding of how to code an assemblyLine. Next, we'll walk through
how you to code similar assemblyLines, using the Connections 3.0 Best Practice model.
8 Connections 3.0 “Best Practice” approach
The Connections 3.0 TDI implementation includes an adapter framework that allows you to write
custom AssemblyLines that can be invoked by the out-of-box (OOB) AssemblyLines.
The adapter concept is a mechanism in TDI that lets developers create new custom Connectors by
using the AssemblyLine (AL) methodology. These Connectors are then invoked from the OOB
AssemblyLines, allowing you to change the default behavior without actually changing the shipped
AssemblyLines. Therefore you can now easily upgrade without have to migrate your custom changes.
In this section we cover the three main Connection AssemblyLines that can be extended via the
adapter framework and show what it takes to actually build an adapter and use it:
collect_dns
populate_from_dns_file
sync_all_dns
88
8.1 Profiles update philosophy
Connections does NOT document the schema due to the fact that as new enhancements are added,
the schema changes; tables may be updated, renamed, deleted, etc. For this reason, updating
Connections tables directly is NOT SUPPORTED or recommended. You should always use existing
assemblyLines such as SyncDBFromSource, new 3.0 Connectors, or the Service Provider Interface.
During testing of your Profiles load, you may want to look at the data in the tables for verification. At
the time of this writing, the relevant tables where data is loaded are as follows:
PEOPLEDB database
Tables:
EMPLOYEE – main employee data table
PROFILE_EXTENSIONS – employee extension attribute data is store here
SURNAME – Surname table
GIVEN_NAME – Givenname table
8.2 Design with upgrade in mind
You should never change the OOB assemblyLines because, once you do this, you cannot easily take
advantage of fixes and enhancements as they become available. To take this one step further, you
should create your own Project, and all custom code should be maintained there.
This way, when you implement new iFixes (new tdisol directories), you can easily move your project
(TDI configuration file) over to the new tdisol directory (see figure 119).
Figure 119. Custom project
89
You can “reference” existing assemblyLines from the OOB configuration, which lets you completely
develop your customizations separate from the OOB assemblyLines:
1.
First, set up a reference to the Connections configuration file as shown in figure 120.
Figure 120. Connections configuration file reference
After you define the reference, you can use it. Note, however, there is a bug in the current
configuration editor such that, after you set the reference, you may need to stop/start the
configuration editor before you can use it in the next step of referencing the existing AL.
2.
Set the Execution Mode field to Manual (cycle mode) on your “referenced” assemblyLine (see
figure 121. Manual mode makes the AL faster since the AL is initialized only once per reference,
limiting the amount of extraneous logging in your ALs.
90
Figure 121. Manual mode
8.3 Project setup
The first thing we want to do is create a new project to contain our custom AssemblyLines. We will
also copy over all the common resources to be used for all our examples.
1.
Start the Configuration Editor. Go to your TDI install directory and issue the ibmditk command,
pointing to your current Connections TDI solutions directory:
2.
Create IBMPoC_Custom_ALs project, using File – New – Project and making sure to select a TDI
project (note the
icon; see figure 122).
Figure 122. New Project
3.
Specify the Project Name, as shown in figure 123; click Finish.
91
Figure 123. Specify project name
4.
Next, we copy over shared resources from our Connections 2.5 example above, using the
IBMPOC properties file along with the IBMPoC_normalizeName script (see figure 124). You can
copy/paste these from your 2.5 sample above to this new project.
Figure 124. IBMPoC_normalizeName script
5.
For the four samples below we use multiple input files and must make sure we have all the
necessary properties in the IBMPoC property file. Here is a list of the necessary input files, all of
which need to exist in your <tdisol> directory:
92
IBMPoCprofiles_tdi.properties. These are the properties we will use to set our connection
properties on our different source connectors:
EmployeeIDs.in. This is the input file for our collect_dns adapter example. It's simply a list of
employee Ids (see figure 125).
Figure 125. EmployeeIDs.in
employeeData.in. This is the input file for our sync_all_dns sample and is also the same input file we
used in the 2.5 example (see figure 126). This is the main driver to the AssemblyLine.
Figure 126. employeeData.in
Now we are ready to start coding our custom AssemblyLines.
8.4 Source repository connectors
By creating a custom source repository connector, you can integrate data from non-LDAP sources
when you are populating Profiles with user data. To integrate a custom source repository, you must
create custom versions of the following two AssemblyLines and then package them as TDI adapters:
•
Iterator connector. This adapter iterates sequentially through all the users in your user directory
and is used by a number of iterative TDI scripts. For example, it's used to retrieve the full
93
information for initial data population via the collect_dns script and during data synchronization via
the sync_all_dns script.
•
Lookup connector. This adapter looks up individual users in your user directory and is used in
the populate_from_dns_file script.
collect_dns. By default this simply goes out to the LDAP and reads all users based on the specified
filter, starting at the specified location (search base). It then writes those users' DNs to a file called
collect.dns.
For most customers this is a great starting point to drive which users need to be populated into the
Profiles database, but occasionally it is necessary to drive the load off something else. That is where
the power of the new adapter framework comes into play.
For our scenario, we occasionally get a list of new employees that need to be added. We don't want to
do a complete sync but only add this set of users. So we want to write an assemblyLine that looks up
a user based on their employee ID versus a default lookup of everyone. We then generate this into an
adapter and let the OOB scripts (collect_dns) use this logic to generate the lookup.
The assemblyLine will be fairly simple: Iterate through a sequential file that contains employee Ids and
then look them up in the LDAP (see figure 127). The collect_dns assembly takes care of counting and
writing the DN to the collect.dns file.
Figure 127. iterate_through_EmployeeIDs
The first thing we need to do is create a new AssemblyLine:
1.
Right-click on the Project name, and select New AssemblyLine (see figure 128).
94
Figure 128. New AssemblyLine
2.
Name the assemblyLine IBMPoC_collect_dns_from_employeeIDs and click Finish (see figure
129). This defines an empty assemblyLine that's ready for us to add our connectors.
Figure 129. Name the AssemblyLine
Now add the feed component. This component (connector) will iterate through our input file:
1.
Right-click on Feed, and select Add Component; or just use the Add component button after Feed
is highlighted (see figure 130).
95
Figure 131. Add component
2.
Select File System Connector, give it a name of iterate_through_EmployeeIDs and a mode of
Iterator, and then click Next (see figure 131).
Figure 131. iterate_through_EmployeeIDs File System Connector
96
Now we define the input file to be used:
1.
Select the Label, in this instance the “File Path” in figure 132.
Figure 132. Insert new object window
2.
In the Expression Editor – File Path window (see figure 133), select the Use Property option,
assign the property collect_input_file (employeeIDs.in), and click OK. This assigns
employeeIDs.in as the input file for our feed.
Figure 133. Expression Editor – File Path window
97
3.
In the next window, select Next (see figure 134).
Figure 134. employeeIDs.in File Path
Now we need to select the Parser for this connector (see figure 135). The parser defines how we will
read this file (whether via a line read, a .csv read, etc.).
Figure 135. Select Parser button
1.
Select Line Reader Parser (we will read one line at a time, with 1 value per line); click Finish (see
figure 136).
98
Figure 136. Select component type
2.
In the next window, click Finish to save the connector (see figure 137).
Figure 137. Line Reader Parser connector
99
3.
To confirm that our connector works, select the Connect button, to connect to the employeeIDs.in
file, and click Next. This reads the first entry. After you verify you can read the first entry, then click
the Close button, to close the connection.
4.
Lastly, add the employeeID to the work attribute: Click Add, to display the Add attribute window,
enter the employeeID attribute, and click OK (see figure 138).
Figure 138. Add attribute window
The window shown in figure 139 displays the completed FileSystemConnector. You can connect to
the resource and verify that you can read the resource correctly.
Figure 139. Completed FileSystemConnector
Next we need to add the logic to lookup the user by adding an LDAP lookup connector to the Data
Flow section:
100
1.
Right-click on Data Flow and select Add Component, or just use the Add component button after
Feed is highlighted (see figure 140).
Figure 140. Add Component
2.
Select LDAP Connector, give it a name of lookup_ldap_user_info and a mode of Lookup, and
click Next (see figure 141).
101
Figure 141. lookup_ldap_user_info LDAP Connector
3.
Define the LDAP property information we will use (see figure 142). Select the Labels; the
Expression Editor window displays.
102
Figure 142. LDAP Connector properties
4.
Change the values for all relevant properties, as shown in figure 143; click OK.
Figure 143. Change values
103
The properties should now be set as shown in figure 144.
Figure 144. Connector Configuration window
Now let's test our Connector and add the $dn work attribute:
1.
First click the Connect button, followed by the Next button; you should see the first record read
(see figure 145).
2.
Click the Close button, to close the connection, and then add the $dn work attribute. This says
that, on a successful read, set the $dn attribute in the work object.
104
Figure 145. Test the Connector and add $dn attribute
3.
Next, set the Link Criteria by clicking the Add button then entering EmployeeNumber for the LDAP
attribute; $employeeID is a work attribute retrieved from the employeeIDs.in file (see figure 146).
Figure 146. Set Link Criteria
Finally, we must add hooks to the lookup_ldap_user_info Connector:
105
Select your lookup_ldap_user_info Connector and then select the Hooks tab (see figure
147). When an employeeID is not found, we want to send a message to the ibmdi.log and
then go read the next entry in the employeeIDs.in file, using this code:
task.logmsg("+++ERROR reading employee: " +
system.skipEntry();
work.getString("employeeID"));
Figure 147. Add hooks to lookup_ldap_user_info Connector
You are now ready to test your AssemblyLine by walking through the AssemblyLine like you did with
the 2.5 sample above. Look in the work object to make sure $dn is correct as you loop through.
Now that you have successfully tested your AssemblyLine, it is ready to be published to an adapter:
1.
Right-click on your AssemblyLine (IBMPoC_collect_dns_from_employeeIDs) and select Publish
(see figure 148).
106
Figure 148. Publish AssemblyLine
2.
Enter the Package ID and specify the File Path to the <tdisol>/package directory (see figure 149).
Note that, depending on your TDI fixpack level, it may be published to the <tdi install>/package
directory. If so, copy it to the <tdisol>/package directory. Click Finish.
Figure 149. Package Information window
107
3.
Once the package is in the <tdisol>/package directory, go to the <tdisol> directory and run the
fixup_tdi_adapters command to correct a TDI bug, as follows:
4.
You must make one more fix to your adapter: Go to the <tdisol>/packages directory and edit your
IBMPOC store, copying the profiles_tdi.properties store and changing the two references
highlighted in figure 150.
Figure 150. Package_IBMPoC_collect_dns_from_employeeIDs.xml file
5.
You must make one more change before you test your connector: In the profiles_tdi.properties
file, add the following property:
source_repository_iterator_assemblyline=Package_IBMPoC_collect_dns_from_employeeIDs:
/AssemblyLines/IBMPoC_collect_dns_from_employeeIDs
This enables the collect_dns AssemblyLine to invoke your connector to build collect.dns.
108
6.
Now it's time to test your connector by running the runal collect_dns command, as follows:
The collect.dns file should now contain DNs for every employeeID entry in the employeeIDs.in file.
populate_from_dns_file
By default this AssemblyLine reads the collect.dns (which can be overridden in the
profiles_tdi.properties file), looks up the user in LDAP, determines manager/secretary based on the
mapping file property, and then updates the Profiles database with the user.
There are many times when you must be able to gather information from multiple data sources, or you
may need to look up the manager ID via a non-standard (something other than UID or DN) method. In
this case you'll extend the populate_from_dns_file AssemblyLine via the new adapter framework.
For our scenario, we look up additional information from a relational datasource and look up the
manager via an employee ID. So we need to write an adapter that overwrites the default data flow.
The AssemblyLine is fairly simple; it looks up (1) LDAP information, (2) additional employee
information in the database, and (3) manager UID via employee ID. Lookup results are put in the work
object, and the populate_from_dns_file AsemblyLine then passes the work object through to
SyncDBfromSource and updates the database.
Instead of walking through creating the AssemblyLine, we can import the IBMPoC_Custom_ALs.xml
download file accompanying this paper and look through the AssemblyLine. So in the AssemblyLine
shown in figure 151, it looks up:
•
user LDAP information based on the DN passed in via the collect.dns (default
populate_from_dns_file passes this in)
•
additional relational information based on the UID from the LDAP lookup above
Currently the DumpWorkEntry script is disabled, but you can enable it to dump out the work object, to
see what your AssemblyLine has set in the work object.
Figure 151. Data Flow folder
109
Publish/fix your assemblyLine (Package_IBMPoC_populate_from_dns_file) into an adapter, as we did
above in the collect_dns example, and then add the following to the profiles_tdi.properties file:
source_repository_lookup_assemblyline=Package_IBMPoC_populate_from_dns_file:/AssemblyLi
nes/IBMPoC_populate_from_dns_file
To do this, run the following:
In the above example there are two records in the collect.dns; it added one record and updated the
other.
sync_all_dns
By default this AssemblyLine adds users if they do not exist, updates users if they do exist, and
deletes/inactivates users if they no longer exist in the LDAP. Occasionally you need to overwrite both
the feed and data flow like we did in the 2.5 example above.
We now turn that AssemblyLine into an adapter and have sync_all_dns use it to drive this process,
thus giving us an easy way to migrate our 2.5 AssemblyLines (with minimal changes) into the 3.0
framework.
We chose this example because it's a very popular scenario. You usually have some type of input file
from a CRM-type system that's usually in the format of a CSV, DB, or other sequential file format type.
Here is the quick review of what we did above in our 2.5 example:
•
In our example, the master data is the HR file, and only users in the HR file should be loaded into
Profiles. So we need to iterate through the HR file and then look up additional information in the
LDAP and database.
•
We know that both the CSV and LDIF have manager information, and that the CSV is the master,
and the LDIF may not be accurate. Since the CSV has the manager employee ID vs. the UID, we
must do a lookup in the LDAP to find the manager's UID.
•
We also notice that, regarding the data, both the first name (givenname) and surname (sn) can
have accented or double-byte characters, so we must be able to look up these users with and
without the accent.
So from a high level we must do the following (see figure 152):
1.
2.
3.
4.
5.
6.
Copy assemblyLine and define necessary resources
Iterate through Master HR file
Look up additional LDAP information
Look up additional database information
Look up manager UID
Normalize names to remove accents (for profile search)
110
Figure 152. Data Flow folder
We must Publish/fix your assemblyLine into an adapter (Package_IBMPoC_populate_from_csv_file)
as we did above in the collect_dns example and then add the following to the profiles_tdi.properties
file:
source_repository_iterator_assemblyline=Package_IBMPoC_populate_from_csv_file:/AssemblyLi
nes/IBMPoC_populate_from_csv_file
Before you run sync_all_dns, it is very important to understand that this command deletes/inactivates
any users who are in the Profiles database who are not in the input file.
Also, make sure you understand all the sync_xxxx parameters in the profiles_tdi.properties, per the
product documentation topic, “Synchronizing LDAP directory changes with Profiles.” Some of the
important properties are as follows:
sync_updates_show_only. Says to show only the changes that will be made, but do not actually
perform the adds/updates/deletes. This should be used for testing your adapter before deletes are
actually performed.
perform_deletion_or_inactivate_for_sync. Controls whether deletes/inactivates are performed
during the sync_all_dns process.
sync_delete_or_inactivate. Controls whether records are deleted or inactivated.
sync_updates_double_check. Controls whether custom delete logic will be executed.
runal sync_all_dns:
In the above example there are 69 records in the employeeData.in file. Due to the fact we have “1”
record being deleted, it means we currently have 70 records in our Profiles database.
NOTE: $dn must be set, even if you do not use it in your mappings. If $dn is not set, the LDAP hash
routine will ignore input records and will act as if there are zero records from your source input.
111
9 Custom delete logic
As part of the sync_all_dns task, Lotus Connections provides a new plug-in point for defining and
using custom TDI AssemblyLines that contain their own delete logic. The delete logic is used to
identify when a user must be deleted from the Profiles database. By default an AssemblyLine is
provided named sync_all_dns_check_if_remove that checks whether a user should be deleted.
You can add custom delete logic without making changes to the existing Connections configuration
file (profiles_tdi.xml). In the example in figure 153, the custom delete logic checks whether the users
“uid” exists in the LDAP. If so, it is assumed that the employee changed organizations, which caused a
DN change in our company. If this is the case, just mark the record for an update instead of deleting it.
Figure 153. Custom delete logic example
The following two properties in the profiles_tdi.properties files control whether the custom delete plugin is used and what it is named:
#sync_updates_double_check=true
#sync_check_if_remove=Package_IBMPoC_custom_delete_logic2:/AssemblyLines/IBMPoC_cust
om_delete_logic
10 New Connectors
Connectors are now provided that allow you to update your Profile resources without having to handle
or know the technical details of the sources you are updating. These are:
ProfileConnector. Use this to retrieve, create, update, and reset profile entries in the Employee,
Profile Extension, and various ancillary employee tables in the Profiles database. The Connector
flattens all these tables into one logical view of the profile data.
112
The ProfileConnector can also be used to reset the status of Profiles and change whether a user
profile is listed as a manager.
PhotoConnector. Use this to retrieve, create, update, and delete photo entries in the Photo table in
the Profiles database. In the sample files accompanying this paper, we use the PhotoConnector in the
IBMPoC_load_photos_from_database AssemblyLine.
PronunciationConnector. Use this to retrieve, create, update, and delete pronunciation entries in the
Pronunciation table in the Profiles database. To see an example of the usage, refer to the
load_pronounce_from_files AssemblyLine.
CodesConnector. Use this to retrieve, create, update, and delete code entries in the various lookaside tables in the Profiles database. For an example of using the CodesConnector you can look at
any of the AssemblyLines used for populating the supplemental tables, such as the populate_country
AssemblyLine.
11 Samples files
The following download files accompany this paper:
IBMPoC_Custom_ALs.xml. This is the configuration file that contains all the samples above. Create
a project and then bring in the configuration file. We walked through this process earlier when we
brought in the main Connections configuration file (profiles_tdi.xml).
IBMPoCprofiles_tdi.properties. This file (along with the default profiles_tdi.properties file) contains
the properties used in these examples. Copy this file into your <TDISOL> directory and modify it to
match your environment.
The configuration file, IBMPoC_Custom_ALs.xml, contains all the examples we used above, in
addition to a couple of other commonly seen scenarios (see figure 154).
113
Figure 154. IBMPoC_Custom_ALs contents
Let's describe the AssemblyLines in more detail.
IBMPoC_collect_dns_from_employeeIDs. This AssemblyLine is fairly simple and is meant to work
as an iterator Connector within the source repository connector. It iterates through a sequential file
that contains employee IDs and then looks them up in the LDAP (see figure 155).
The collect_dns assembly counts and writes the DN to the collect.dns file and is the assembly
mentioned above in the collect_dns section.
Figure 155. IBMPoC_collect_dns_from_employeeIDs
The input file is specified in the IBMPoCprofiles_tdi.properties files in the samples file. It is controlled
by the property, collect_input_file=employeeIDs.in.
If you want to test this, simply create a sequential file with a list of keys. If it is something besides the
LDAP employeeNumber, you will need to change the link criteria in the lookup_ldap_user_info
Connector.
IBMPoC_custom_delete_logic. This AssemblyLine does an extra check to determine whether a
record actually should be deleted when sync_all_dns is run.
114
IBMPoC_load_photos_from_database. This AssemblyLine gives you an example for reading a
photo from a source database table and then uses the PhotoConnector to update the PHOTO table.
IBMPoC_load_photos_from_database_extra_mapping_code. This AssemblyLine is the previous
one, except it included an extra script in case you need to convert the image into a byte array.
IBMPoC_populate_from_dns_file. This assemblyLine is fairly simple and is meant to work as a
lookup Connector within the source repository Connector (see figure 156). This is the assemblyLine
discussed above in the populate_from_dns_file section.
Figure 156. IBMPoC_populate_from_dns_file
It looks up user LDAP information based on the DN passed in via the collect.dns (default
populate_from_dns_file passes this in), and then looks up additional relational information based on
the UID from the LDAP lookup above. All the returned information is placed in the work object, after
which the source repository Connector passes all the data to the SyncDBfromSource AssemblyLine.
IBMPoC_populate_from_csv_file. This AssemblyLine is a little more complex but is quite typical of
many customers' environments. It's meant to work as an iterator Connector within the source
repository connector (see figure 157). This is the AssemblyLine discussed above in the sync_all_dns
section.
Figure 157. IBMPoC_populate_from_csv_file
In this typical scenario, we have an HR sequential file as our input that controls who is loaded in the
Profiles database. Instead of using the LDAP to determine who is loaded, it uses the HR file.
IBMPoC_update_delta_viaSynch. This is a typical sample for synchronizing Profiles based on a
delta table. The input is usually in the form of a CSV or database table. In our example, we use a
database table that contains an action code to either update/add or delete the record (see figure 158).
115
Figure 158. Database table
This example reads a delta record, checks the the action, and then calls SyncDBfromSource to do the
work.
IBMPoC_update_delta_viaConnector. This sample is quite similar to the previous one, except it is
using the ProfileConnector to do the work. This sample is not a best practice; rather, it is mainly a
sample to show the use of the ProfileConnector. The one thing to keep in mind is that the
ProfileConnector does not use the map_dbrepos_from_source file mappings.
12 Conclusion
Lotus Connections 3.0 uses the robustness of TDI to bring your important person data into Profiles. In
version 3.0, you can now implement a “best practice” approach to extending the OOB AssemblyLines,
allowing you to easily upgrade your custom solutions without having to re-integrate your custom code
with each upgrade/ifix.
13 Resources
•
Refer to the Lotus Connections wiki.
•
Refer to the Tivoli Directory Integrator information center.
•
Read the Tivoli Directory Integrator Getting Started Guide.
•
Take the Tivoli Directory Integrator tutorial, “Handling Exceptions/Errors with TDI.”
•
Watch the videos, “TDI is alive" and TDI on YouTube.
14 About the authors
Chris Biega is a Senior Certified Client Technical Professional for the Americas. He is a member of
the USIMT Lotus Solutions team and specializes in the Lotus Connections product. He can be
reached at [email protected].
Charlie Price provides IBM Level 2 support for Lotus Connections and IBM Lotus Domino Portal
Integration. He is a frequent speaker at Lotusphere® and the Portal Excellence Conference. Charlie
has also authored numerous articles in IBM Redbooks® publications, white papers, and technotes.
He can be reached at [email protected]
Acknowledgement
The authors extend a special thanks to Mike Ahern and Jackie Ferguson who answered endless
questions during the writing of this paper.
116
Trademarks
•
DB2, developerWorks, Domino, IBM, Lotus, Lotusphere, Redbooks, Tivoli, and WebSphere are trademarks
or registered trademarks of IBM Corporation in the United States, other countries, or both.
•
Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States, other
countries, or both.
•
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun
Microsystems, Inc., in the United States, other countries, or both.
•
Other company, product, and service names may be trademarks or service marks of others.
117