Bulk processing in the DBQueue Processor
Table 187: Configuration parameter for bulk processing in the DBQueue Processor
|
QBM | DBQueue | DefaultRuntime |
The configuration parameter species how the length of the DBQueue Processor run. The default value is 90 seconds. |
|
QBM | DBQueue | ChangeLimitMin |
The configuration parameter defines the lower limit for modifications (insert, change, or delete) within a single operation. The default value is 3000. |
|
QBM | DBQueue | ChangeLimitMax |
The configuration parameter defines the upper limit for modifications (insert, change, or delete) within a single operation. The default value is 50000. |
Some DBQueue Processor procedures are marked for bulk processing to reduce the total time required for processing DBQueue tasks. If a lot of entries are marked for bulk processing in the DBQueue, the DBQueue Processor switches from single to bulk processing.
There is a mechanism implemented that is used to decide whether switching to bulk processing as opposed to single processing would result in time savings. To do this, 25 single task processes are run and the processing time is recorded. All other entries for the task are processed in bulk and the minimum and maximum load time required for advantageous bulk processing is defined. A self optimizing calculation procedure updates the load times. Use of this method means that the DBQueue Processor must first stabilize, especially after an initial schema installation or after system modifications such as memory expansion in the database server.
You can use the QBM | DBQueue | DefaultRuntime configuration parameter to specify the length of the DBQueue Processor run. The default value is 90 seconds. This corresponds to the time period that achieves the best load for the calculation procedure.
To prevent overloading when there is large amount of data, you can define limits for the result set. Control is realized using the QBM | DBQueue | ChangeLimitMin and QBM | DBQueue | ChangeLimitMax configuration parameters.
Processing DBQueue Processor tasks by the Database Agent Service
To process internal tasks by the Database Agent Service, ensure that the Database Agent Service is installed and configured. The Database Agent Service is deployed through the One Identity Manager Service plug-in. Alternatively, the Database Agent Service can be run from the DatabaseAgentServiceCmd.exe command line program. The plug-in should be configured on the Job server that performs the Update server server function. An administrative user must be used for the database connection in the Job provider.
The Database Agent Service carries out the following tasks:
-
Controls processing and distribution of the DBQueue tasks to the individual slots (central dispatcher))
-
Checks and starts scheduled tasks
-
Removes already processed DBQueue tasks
-
Starts maintenance work
-
Populates and updates the Job queue overview
-
Archives or deletes handled processes from the Job queue
-
Archives or deletes process handling logs
NOTE: If the Database Agent Service is not working, a message is displayed in the status bar in all the administration tools. To see this message, users must have at least the configuration user access level.
Related topics
Controlling processing of DBQueue Processor tasks
The central dispatcher assumes control of processing and distributes DBQueue tasks to individual slots.
First, it determines the number of currently available slots available for use. The more load there is on the database, the less slots there are to use. However, at least five slots are used.
The number of currently available slots results from:
The number of currently available slots = maximum number of available slots - sum of all own database processes - sum of processes of other databases on the server
The central dispatcher starts the currently available slots just once. or each slot, a process is set up that runs tasks for exactly this slot.
Once tasks in the DBQueue are entered, the central dispatcher is notified. The central dispatcher distributes tasks to individual slots and notifies the slot processes that there are tasks are waiting to be processed. Each process processes the tasks queued for its slot. Once the task is complete, each process sends a message to the central dispatcher and waits for new tasks.
The central dispatcher checks at defined intervals whether the slots are still active and distributes new tasks to them. If there are no more tasks in the DBQueue, the central dispatcher goes into a wait state and waits for new task notifications.
Tasks on slots currently in use are still processed when the timeout expires. On completion, the slots are stopped. For more information, see How the central dispatcher communicates with individual slots.
Processing DBQueue Processor tasks
The central dispatcher finds entries in the DBQueue (DialogDBQueue table) and moves the tasks into the QBMDBQueueCurrent table with the assignment tasks per slot.
Example of entries in the DialogDBQueue and QBMDBQueueCurrent tables
Table 188: Entries in the DialogDBQueue (extract) table
|
OrgRoot |
A |
|
OrgRoot |
B |
|
ADSAccountInADSGroup |
X |
|
ADSAccountInADSGroup |
Y |
|
ADSAccountInADSGroup |
Z |
Table 189: Entries in the QBMDBQueueCurrent (extract) table
|
001 |
OrgRoot |
A |
|
001 |
OrgRoot |
B |
|
002 |
ADSAccountInADSGroup |
X |
|
002 |
ADSAccountInADSGroup |
Y |
|
002 |
ADSAccountInADSGroup |
Z |
Each process processes tasks queued for its own slot in the QBMDBQueueCurrent table. Subsequent tasks resulting from processing are queued in the DialogDBQueue table.
If a process has processed its tasks and no other tasks are pending, the slot number in the QBMDBQueueCurrent table is set to 0 by the process itself. The entry initially remains in the QBMDBQueueCurrent table but is no longer taken into account (because slot 0 is not active).
All entries with the slot number 0 are deleted from the QBMDBQueueCurrent table at regular intervals.
Table 190: Meaning of slot numbers in the QBMDBQueueCurrent table
|
001 - n |
Number of slot to be processed by the task. |
|
0 |
State after the task is completed correctly. |
|
-1 |
An error occurred during task processing or processing was deferred, for example, because synchronization is running. The central dispatcher re-enables the task.
NOTE: Deferring DBQueue tasks is recorded in the system journal. |
|
-2 |
An error occurred during task processing or processing was deferred, for example, because of blocking. The central dispatcher re-enables the task. |
|
-3 |
An error occurred during task processing or processing was deferred, for example, because there are still entries in the Job queue. The central dispatcher re-enables the task. |
Using the DBQueue buffer
To prevent blockages when processing DBQueue tasks by lengthy actions, for example, synchronization, a DBQueue buffer (QBMDBQueuePond table) is used. Synchronization initially writes DBQueue Processor tasks to the QBMDBQueuePond table. After synchronization is complete, the task are moved from the QBMDBQueuePond table to the DialogDBQueue table.
If a lengthy task does not queue anymore entries in the DBQueue tasks because, for example, synchronization did not end correctly, the remaining entries in the QBMDBQueuePond table are moved to the .DialogDBQueue table. The time period for this is defined in the QBM | DBQueue | BufferTimeout configuration parameter (default: 120 minutes). The transfer is carried out by the daily maintenance tasks.
