Installation standalone

Standalone distributions / products come with a predefined set of jadice flow components and workers to achieve specific use cases. In theory, almost every jadice flow product could also be shipped as standalone product. Since the main deployment platform is kubernetes, this is not always the case by default.

Products might provide more specific installation instructions. This documentation gives a general overview of a standalone product.

Package contents

This package contains:

Main controller and worker components:

• jf-controller: Contains the jadice flow controller. This is the main component controlling jobs and managing the workflows.
• jf-controller-ui: Contains the UI. This is a web based UI to manage the controller
• worker-*: Contains specific worker applications, e.g. worker-topdf

Additional components

• jre: Jadice flow comes with a bundled JRE (Java 17).
• mariadb (optional): MariaDB installation package is provided for convenience. This is an optional but recommended installation, since the H2 database will be used otherwise (not recommended in production). See installation chapter for more info.
• minio: (Scripts only) The S3 data storage - please see the specific storage chapter
• wrapper: Contains the java service wrapper which is used to start the java applications
• ZMC: (optional) Contains the Azul Zulu Mission Control. This is a diagnostic tool which can be used to attach to the services and read application metrics via JMX.

Prerequisites

Currently, the only prerequisite is a S3 compatible storage.

The AIO can be found at https://levigo.de/maven2. Access to this repository can be granted on request. Please contact jadice-support@levigo.de.

Configuration of products components

Extract the ZIP into a folder. The extracted folder contains a sub-folder for each component. In the root folder, scripts to install or run the services are available. The install-all-services-Script installs the main components. Additional components have to be installed seperately.

All components which require a configuration have a separate config subfolder which might contain further or specific configuration.

jf-controller

The jadice flow controller has the following configuration files:

• application.yaml: Main configuration file with server port etc.
• workers.yaml: Configuration file which defines which workers are available (this file is usually pre-configured and does not need to be changed)
• jobtemplates.yaml: Configuration file for job templates. Pre-configured with a default template. This template could be checked for parameter "placeholders". Either you can edit the file directly or use the controller-UI to check the configuration in the browser (recommended).
• wrapper.conf: Java Service Wrapper configuration file (pre-configured)

For more information see General configuration

jf-controller-ui

The jadice flow controller-ui has the following configuration files:

• application.yaml: Main configuration file with server port etc.
• wrapper.conf: Java Service Wrapper configuration file (pre-configured)

worker-*

The workers have in general the following configuration files:

• application.yaml: Main configuration file with server port etc.
• wrapper.conf: Java Service Wrapper configuration file (pre-configured)

Some workers may have additional files for configuration. Some workers also might provide a native binary instead of the wrapper.

Initial configuration to modify before start

Database

The jadice flow controller utilizes a jdbc database to store job runtime information. By default, the standalone distribution will use a H2 in memory runtime Database (DB) for testing / showcasing purposes.

For production, an external DB should be used to keep job information persistent. This product comes with a mariadb package which can be installed locally if needed.

Local MariaDB installation

By default, the in memory H2 DB is being used.

To use the maria db, the application.yaml file of the jf-controller-component needs to be modified:

  # JDBC Config in application.yaml
  datasource:
    initialization-mode: always
    url: jdbc:mariadb://localhost:3306/jadice_flow?createDatabaseIfNotExist=true
    username: jadice-flow
    password: jadice-flow
    hikari:
      schema: JADICE_FLOW
      maximum-pool-size: 50

s3 storage

The application.yaml files are pre-configured to use a local s3 with the following configuration:

publisher:
  s3:
    bucket: jadice-flow-bucket
    endpoint: localhost:9000
    access-key: jadice-flow
    secret-key: jadice-flow
    protocol: http

If an external S3 shall be used, the application.yaml files have to be adjusted accordingly.

Local S3 using MinIO

One possible local S3 solution is MinIO. This open source S3 implementation is licensed under the AGPL-License. For this reason, MinIO does not come bundled with this package.

Please check the minio folder for a setup guide: minio-installation.html.

Alternatives for local S3 storage:

• MinIO
• Ceph -

This (external) ceph installation tutorial shows the basic steps required to run ceph locally.

Neverpile Eureka

Neverpile Eureka can be used to store job results. It requires zero configuration for test installations. It's also possible to enabling persistence and configuring other storage backends.

By default, the expiration time of saved job results is 3 minutes. The expiryTimeMinutes can be configured in the file <bundle>/neverpile-eureka/config/application.yaml:

neverpile-eureka:
  storage:
    ehcache:
      enabled: true
#      rootPath: ${user.dir}/neverpile-eureka_default
#      heapEntries: 500
#      diskSize: 1024
      expiryTimeMinutes: 5

Usage

The components can be installed as service, or run directly on the command line. Several scripts are provided for these cases:

• install-all-services: Installs the main components as service. Additional components like mariadb have to be installed seperately via separate single install scripts, e.g. install-mariadb.bat.
• uninstall-all-services: Uninstalls all services which were installed via install-all
• The start-all-services and stop-all-services scripts can be used to start and stop the services

If no service installation is needed, the components can be started directly:

• run-jf-controller-console
• run-jf-controller-ui-console
• run-worker-* : Here with a wildcard - there is a run script for each provided worker

Docker

In the following chapter you can find all information to run and build an AIO Docker Image in one Docker container.

Prerequisite

Beside the prerequisite for the AIO, for some Bundles are some additional installations in the container image necessary:

Email Conversion:

For the Email Conversion Bundle are the following installations in the container image necessary. You can use the AIO Base Image to install this dependencies:

• libreoffice
• imagemagick
• tesseract

Note: This jadice flow Bundle uses ImageMagick Studio LLC to convert Heic/Heif images. For more information see: ImageMagick software

Build an AIO Docker Image

Use Docker Compose

Each bundle has a docker-compose.yaml to build an image which contains the AIO package to run it in one container. To Build and run an image, run in the bundle root directory the following command:

$ docker compose build

Use Dockerfiles

Alternatively to Docker Compose each bundle has also a Dockerfile to build a Docker image which contains the AIO package. To Build an image, go into the bundle root directory and run the following commands:

Build the base image:

$ docker build -t aio-base:1.1.5 -f Dockerfile_base .

Build the application image:

$ docker build -t jf-aio/jf-email-conversion:1.0.0 .

To run the AIO Docker Image use docker compose run or use the appropriate docker run commands.

Logging

By using the run-all-console.sh script, all logs of the jadice flow components are logged to the console. To log the respective logs of a jadice flow component into a corresponding console.log file, you can use the run-docker.sh script. After that, you can find the console.log files in the worker folders.