Skip to main content
Version: Next

Controller configuration

The jadice flow controller is a spring boot application and can run in a cloud environment or on a local machine. It is the main component controlling jobs and managing the workflows.

Kubernetes / Cloud environment is configured via the helm chart. In the standalone distribution, the configuration files reside in the corresponding /config-directories.

application.yaml​

The application.yaml is the main configuration file. It contains all configuration aspects of the spring boot application itself (e.g. port, keycloak-enabled, metrics configuration).

JobTemplates are stored in a separate file called jobTemplates.yaml. This allows the controller to persist only the JobTemplates, when the configuration has changed (e.g. via the UI). The jobTemplates.yaml is imported into the application.yaml via an import statement. It is possible to omit the jobTemplates.yaml and directly place the JobTemplates in the main configuration file, but this is not recommended.

When executing a JobTemplate, this is done in a Job. Each step of the JobTemplate might call an external jadice flow worker.

The possible workers are configured in the workers.yaml file. It is imported into the main configuration file like the jobTemplates.yaml. The controller will not persist / change the worker configuration though; only job templates can be configured dynamically (e.g. via UI).

Database​

In a standalone distribution, the jdbc source is configured in the application.yaml file of the controller directly:

spring:
  datasource:
    url: jdbc:mariadb://localhost:3306/jadice_flow?createDatabaseIfNotExist=true
    username: jf_db_user
    password: password
    hikari:
      schema: JADICE_FLOW
      maximum-pool-size: 50

Jadice flow main configuration​

Main configuration:

# Jadice flow main config
jadice-flow:
  server-url: http://localhost:8080/
  securityToken: _token_
# System config / Scaling
  system:
    lockJobConfiguration: false
    configFileJobs: ${user.dir}/config/jobtemplates.yaml
    maximumConcurrentJobCount: 4
    maximumConcurrentWorkerInvocationsDefault: 1
    ## the number of parallel invocations can be configured per worker. Use the 'workerName' from the workers.yaml
    #maximumConcurrentWorkerInvocationsSpecific:
    # single-image-to-tiff: 3
  jobDeletion:
    enabled: true
    ## see https://docs.spring.io/spring-framework/reference/integration/scheduling.html#scheduling-cron-expression
    ## Run the job every weekday at 4:05 AM.
    #cronSchedule: "0 5 4 * * MON-FRI"
    ## Run once every minute.
    #cronSchedule: "0 */1 * * * *"
    ## Run once every eight hours.
    cronSchedule: "0 * */8 * * *"
    # Jobs will be deleted in batches of this size
    maxDeleteRequestSize: 500
    # ThreadPool of the job deletion - the batches are split between the threads.
    threadPoolSize: 5
    # Minimum Age of the job
    thresholdTime: 12h
    mode: ALL_PARTS # one of ALL_PARTS, NO_PARTS, ONLY_META_PARTS, ONLY_BASE_PARTS
    exitStatus:
      - COMPLETED
    #      - FAILED
    #      - EXECUTING
    #      - STOPPED
    #      - NOOP
    #      - UNKNOWN
    jobDeletionFilters: []
    # example:
    #  - "[{\"type\":\"PartPropertyFilter\",\"inclusive\":true,\"propertyKey\":\"created-by\",\"matcher\": { \"type\":\"StringMatcher\",\"type\":\"StringMatcher\",\"mode\":\"EQUALS\",\"string\":\"ToPDF\",\"ignoreCase\":true } }]"
  event-bus:
    enabled: true

Most noteworthy configuration parameters:

  • jadice-flow.system.lockJobConfiguration: Allows to lock the system configuration so it cannot be changed via Rest / UI
  • jadice-flow.system.configFileJobs: The available JobTemplates are imported into the application.yaml from the jobtemplates.yaml by default.
  • jadice-flow.system.maximumConcurrentJobCount: Defines how many jobs may run in parallel

Automatic job deletion​

The automatic recurring deletion of jobs is enabled by default.

  • You can configure when the deletion shall run as a Spring Cron Expression with cronSchedule. It can also be triggered via REST.
  • how old a job has to be in order to be deleted threshold-time
  • what kind of parts of each job are deleted mode. Available values: ALL_PARTS, NO_PARTS, ONLY_META_PARTS, ONLY_BASE_PARTS
  • what exit-status a job must have to be deleted exit-status. Available values: COMPLETED, FAILED, EXECUTING, STOPPED, NOOP, UNKNOWN

Trigger job deletion manually​

With the following curl command you are able to trigger the jobdeletion manually. The jobDeletionFilters will be ignored.

curl -X POST --location "http://jf-controller.acme.com/admin/trigger-job-deletion" \
    -H "Content-Type: application/json" \
    -d 'true'

License​

To use the flow controller, a license is needed. The license which is provided is added to the file as a separate block:

jadice:
  license-configuration:
    license: |
      ----BEGIN LICENSE----
      Ic5OXgAAACQAAAACAAAACAAAABBjdXN0b21lcnsibmFtZSI6IlRSSUFMIn0AAAAa
      ....
      c2lnbmF0dXJlRGlnZXN0U0hBLTUxMg==
      ----END LICENSE----
    fingerprint: 6149d57c-...
    public-key: |
      -----BEGIN PUBLIC KEY-----
      MIICITANBgkqhkiG9w0BAQEFAAOCAg4AMIICCQKCAgANW2cb2XJxgnysASNpfQfL
      ...
      iLjj/Yso3fK+SkDr0ZVMqQIDAQAB
      -----END PUBLIC KEY-----

Auto job restart​

The controller can automatically re-start jobs if they fail. This can be useful in the following use cases:

  • The worker was temporarily unavailable, resulting in an I/O error message
  • If the worker use external systems (e.g. the cm8 workers, or s3 output), a restart can be useful on temporary problems

To enable the auto restart, add the following block to the application.yml of the controller:

jadice-flow:
  
  # ...

  # Job auto restart configuration
  auto-restart:
    enableOnAppRestart: true
    enableCyclicCheck: true
    enableEventBus: false
    cyclicCheckInterval: 5m
    minJobAge: 5m
    jobExitCodes:
    - "FAILED"
    # Exit codes only for startup check. If empty, jobExitCodes is used.
    jobExitCodesStartup:
    - "FAILED"
    - "STARTED"
    - "STARTING"
    maxRestartCount: 2
    messageFilter:
    # Example Error message produced by "local-sleep" Processor, if "failPercent"-Property is set
    - "Processing failed.*"
    # Message filter only for startup check. If empty, messageFilter is used.
    messageFilterStartup:
    # Caution: ".*" Wildcard could be used to start all jobs regardless of the message.
    # A more specific message is still preferred (or empty to use the messageFilter)
    - ".*"

Description of jadice-flow.auto-restart Properties:

  • enableOnAppRestart - if enabled, FAILED jobs will be checked on controller start. Recommended to be enabled. There are optional jobExitCodesStartup and messageFilterStartup settings which can be set only for the startup phase. For example, in the cyclic check we do not need to restart currently running jobs, but on application restart we want to restart Jobs still in STARTED state.
  • enableCyclicCheck - if enabled, jobs will be checked regularly if restart is required. Recommended.
  • enableEventBus - if enabled, FAILED jobs will be checked on receiving the jobEvent. This in combination with "minJobAge" set to 0 would allow an instant job restart. But in most cases, the event bus can be disabled to rely on the cyclic check.
  • cyclicCheckInterval - the interval in which to check FAILED jobs for restart
  • minJobAge - the minimum age of a job to be considered for restart (time between last end date and now)
  • jobExitCodes - the job exit codes to be restarted, by default only "FAILED"
  • jobExitCodesStartup - only for the startup check: the job exit codes to be restarted, by default "FAILED", "STARTED", "STARTING" (to also restart previously running jobs). If the list is empty, the jobExitCodes list is used.
  • maxRestartCount - the maximum auto restart count for a job. If still failed, it wont be restarted later.
  • messageFilter - a (optional) list of messages (or regex) which trigger the auto restart. It is recommended to enter the desired message here upon which a restart shall take place. Some errors are persistent (e.g. document conversion errors), but network specific problems are good candidates for the list. (e.g. feign.RetryableException.*). Regex supported, but the more specific, the better. The list of job messages consist of the step execution and item execution log messages, so if an error is shown in the UI somewhere for example, that message is also available for filtering here (AuditInformation from Job).
  • messageFilterStartup - an (optional) separate message filter only for the startup check. If empty, the normal message filter is used.

The message filter is optional, but strongly recommended to avoid restarting on permanent errors.

Note: The current auto-restart-count value is stored in the job properties of the jadice flow job. In the job properties (map with input properties), there is also a json entry with key processingProperties which contain the real jobs runtime processing properties, including the restart counter.

jobTemplates.yaml​

As mentioned before, the jobTemplates.yaml contains the workflow definitions.

Note: The amount of active job templates might be restricted by the license. If more job templates are available than the license permits, some templates have to be disabled.

Example content:

---
jadice-flow.jobs:
  jobTemplates:
  - jobName: "Local-Sleep"
    description: "A local test template which sleeps for each item."
    properties: {}
    enabled: true
    stepTemplates:
    - stepName: "Sleep"
      workerDefinitionName: "Sleep"
      inputMimeTypes: []
      expectsNewPartResult: false
      markSrcAsMetaOnResult: false
      parameters: []
    jobFlow:
    - from: ""
      "on": "*"
      to: "Sleep"

This file should only be edited manually by advanced users. To configure JobTemplates, users should generally use the UI / Rest endpoints.

workers.yaml​

The workers.yaml contains the worker definitions of possible callable workers.

Note When using the auto-discovery feature (see Event bus configuration), no workers.yaml file is required. Workers will auto-register themselves at the controller.

This section describes how to configure the workers.yaml manually, if no auto-discovery is active.

Example:

---
jadice-flow.workers:
  workerDefinitions:
    - workerName: "uuid"
      description: "Creates a uuid in the property NEW_UUID"
      processorClass: "com.jadice.flow.controller.server.processor.RemotePartProcessor"
      workerURL: "https://jf-worker.net/"
      infoTags:
       - "PART_BASED"
       - "REMOTE"
      workerParameters: []

Usually, jadice flow will come with a pre-configured workers.yaml with the relevant workers already entered for the customer.

Main parameter is the workerURL, the other settings define how the controller shall interact with the worker (the processor class) and which description to use. Those values usually do not need to be changed by the customer.

There might be the need to customize the worker-URLs to fit the current environment.

Last updated on by David Hoffmann