Überleben im OSB/SOA Dschungel Daniel Joray

2015 © Trivadis

BASEL BERN BRUGG GENF LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

KOPENHAGEN

2015 © Trivadis

Überleben im OSB/SOA-Dschungel

Daniel Joray

2015 © Trivadis

Trivadis ist führend bei der IT-Beratung, der Systemintegration,

dem Solution-Engineering und der Erbringung von IT-Services

mit Fokussierung auf und Technologien

im D-A-CH-Raum.

Unsere Leistungen erbringen wir aus den strategischen Geschäftsfeldern:

Trivadis Services übernimmt den korrespondierenden Betrieb

Ihrer IT Systeme.

Unser Unternehmen

20.11.2015

Trivadis – das Unternehmen

B E T R I E B

2

2015 © Trivadis

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort

3

12 Trivadis Niederlassungen mit

über 600 Mitarbeitenden

200 Service Level Agreements

Mehr als 4'000 Trainingsteilnehmer

Forschungs- und Entwicklungs-

budget: CHF 5.0 Mio. / EUR 4.0

Mio.

Finanziell unabhängig und

nachhaltig profitabel

Erfahrung aus mehr als 1'900

Projekten pro Jahr bei über 800

Kunden

Stand 12/2013

Hamburg

Düsseldorf

Frankfurt

FreiburgMünchen

Wien

Basel

ZürichBern

Lausanne

3

Stuttgart

20.11.2015


3

Brugg

2015 © Trivadis


Trivadis an der DOAG

Ebene 3 - gleich neben der Rolltreppe

Wir freuen uns auf Ihren Besuch.

Denn mit Trivadis gewinnen Sie immer.

2015 © Trivadis

AGENDA

1. SOA & OSB Project

2. Installation

3. WLS Domain Configuration

4. OSB Cluster

5. SOA Cluster

6. SOA Schema

7. Testing

8. Summary


2015 © Trivadis


About SOA & OSB

2015 © Trivadis

SOA & OSB Project – General


The Oracle SOA Suite is a comprehensive, standards-based software

suite to build, deploy and manage integration following the concepts of

service-oriented architecture(SOA)

The Oracle Service Bus is an enterprise service bus (ESB) that

provides the key virtualization layer required for any sustainable

integration architecture

2015 © Trivadis

SOA & OSB Project – Focus


External hosted applications can execute service requests

to the main application

The internal GUI application should use the same services

to retrieve the data.

Authentication is done before in a “secure zone” - a token

is delivered with the request

Coherence will be use to Cache Data between the OSB,

SOA and GUI components

2015 © Trivadis

SOA & OSB Project – Requirements

Application should be high available

Application should support 200 requests per second

98% transactions should be faster than 600 milliseconds

Database Size is about 1 TB

The Application should deliver data to subsystems e.g. for a

batch job


2015 © Trivadis

SOA & OSB Project – The Architecture


2015 © Trivadis

SOA & OSB Project – Lessons learned


In the next slides, you will see our

lessons learned

2015 © Trivadis


Installation

2015 © Trivadis

Installation - General

Use distinct middleware home for SOA and for OSB.

Install your own JDK, don’t use the system JDK

In this project, we got better performance with Oracle JDK as with

JRockit


2015 © Trivadis

Installation – Patching

Install the last patch set and PSU in comparison with the

base version Patch 20423630: OSB Bundle Patch 11.1.1.7.4 fix 118 bugs

Patch 20900797: SOA Bundle Patch 11.1.1.7.8 fix 768 bugs

Patch 20780171: SU Patch [EJUW]: WLS PATCH SET UPDATE

10.3.6.0.12 fix more than 700 bugs

Nobody will spend time for upgrade in a project phase, but

this is very important to plan updates regularly in the

project

Use a recent JDK compatible with your middleware

version

After each patch, regression testing must be done


2015 © Trivadis


WebLogic Domain Configuration

2015 © Trivadis

WLS Domain Configuration – General

Create your domain in Production Mode to maximize performance

For performance testing and production environments, consider using

the lowest acceptable logging level, such as "ERROR" or

"WARNING" whenever possible

Consider turning off the HTTP access logging

If You use non-persistent JMS scenarios, explicitly turn off persistence

at the JMS server level - for persistent JMS scenarios you can use file

store or JDBC store

If there are multiple JMS servers involved, create each JMS store on

a separate disk to lower I/O contention


2015 © Trivadis

WLS Domain Configuration – JVM Memory

The out-of the box JVM configuration provided by the default

installation of OSB, SOA or WebLogic Domain is probably not optimal

for your application (-Xms512m –Xmx1024m)

Not only the Heap Size should be tuned, take care for the ratio

between Young and Tenured

Verify the Garbage Collection algorithm

Sample of the memory argument from the OSB Managed Server


-Xms8192M -Xmx8192M -XX:+UseParNewGC -

XX:+UseConcMarkSweepGC -XX:NewSize=4096m -

XX:MaxNewSize=4096m -XX:SurvivorRatio=6 -

XX:TargetSurvivorRatio=90 -XX:PermSize=512m -

XX:MaxPermSize=768m -XX:ReservedCodeCacheSize=120m

2015 © Trivadis

WLS Domain Configuration – Connection Pools


Configure the Connection Pools

Ensure that the connection pool has

enough free connections.

Set Initial Capacity and Minimum

Capacity to the same value

Disable unnecessary connection testing

and profiling

Enabling Data Source Statement

Caching

Take care with the Option “Wrap

datatypes” and DB Adapter

2015 © Trivadis

WLS Domain Configuration – TCP

The Accept Backlog parameter specifies how many Transmission

Control Protocol (TCP) connections can be buffered in a wait queue.

Tune the Default Work Manager.

The Default WorkManager is used to handle

thread management and perform self-tuning.


-DWebLogicic.threadpool.MinPoolSize=100

-DWebLogicic.threadpool.MaxPoolSize=250

2015 © Trivadis

WLS Domain Configuration – Work Manager

For OSB you can create Work Manager in the WebLogic Domain


2015 © Trivadis

WLS Domain Configuration – Time Zone / Archiving

Force WLS to use the right time zone

Set Configuration Archive Enable on True and set the Archive

Configuration parameter


-Duser.timezone=Europe/Zurich

2015 © Trivadis

WLS Domain Configuration – Cluster or not Cluster

If your system must be permanently available, it is best to define a

cluster configuration type from the beginning of your project

SOA and OSB environments are extremely slow to start (several

minutes) which makes them unusable failover type configuration

A cluster type of configuration is easily modified to accept more load

depending on demand.

For SOA and OSB cluster configurations are robust even under high

load variation or if some managed server are not available


2015 © Trivadis

WLS Domain Configuration – Oracle HTTP Server

If you use mod_wl_ohs from the Oracle HTTP Server or mod_wl for

Apache, set the parameter WebLogic Plug-In Enabled to true


2015 © Trivadis


OSB Cluster

2015 © Trivadis

OSB Cluster – Proxy Service Caching

Oracle Service Bus 11g provides built-in cache for proxy services may

help throughput (Compiled Proxy Cache )

This cache management use algorithm (LRU)

The default value is 100 and can be increased for large number of proxy

services

If you deploy a large numbers of services in your OSB, you should

increase the size of this cache


-Dcom.bea.wli.sb.pipeline.RouterRuntimeCache.size=700

-Dcom.bea.wli.sb.pipeline.RouterRuntimeCache.preload=true

2015 © Trivadis

OSB Cluster – Result Caching (1)

For result caching, OSB provides built-in cache functionality that uses

Oracle Coherence

Per default, the Coherence Cache is part of the OSB Managed Server

OSB Cache Configuration is located unter$DOMAIN_HOME/config/osb/coherence/

Coherence Cache Configration is defined in file osb-coherence-

cache-config.xml

Coherence Network Configuration is define in file osb-coherence-

override.xml

If you want no spend OSB JVM Memory for the cache, you can

configure external Coherence Server and change the local storage for each cache


2015 © Trivadis

OSB Cluster – Result Caching (2)

Result caching definition is done on service level

You can enable or disable result caching for the OSB Domain in the console


2015 © Trivadis

OSB Cluster – Load Balancing Outgoing Request

(HTTP)

Specify Load Balancing Algorithm and list of Endpoint URIs

3 settings that help with endpoint failures

Retry Count - Retry Iteration Interval - Retry Application Errors

You can use an external load balancer


2015 © Trivadis

OSB Cluster – Load Balancing Outgoing Request

(JMS)

For JMS you have the same possibilities, but the syntax is a quite

different


2015 © Trivadis


SOA Cluster

2015 © Trivadis

SOA Cluster – Post Configuration (1)

Oracle SOA Suite 11g use an embedded Coherence cache to

coordinate several cluster-wide activities including composite

deployment

By default, this embedded Coherence cache is configured for

multicast node discovery

To allow the communication between the coherence cache members,

you should add in the setDomain.sh/cmd these properties

In setDomain from Managed Server 1

In setDomain from Managed Server 2


-Dtangosol.coherence.wka1=“ServerName1”


-Dtangosol.coherence.localhost=“ServerName1”



-Dtangosol.coherence.localhost=“ServerName2”

2015 © Trivadis

SOA Cluster – Post Configuration (2)

Configure the Java Object Cache (JOC) procedure is optional, but

increases the performance of Oracle WSM by keeping a local cache

instead of having to search for a cache.

Perform the 'soa-createUDD.py' to create the JMS Cluster


$JAVA_HOME/bin/java weblogic.WLST

$MW_HOME/oracle_common/bin/configure-joc.py

$JAVA_HOME/bin/java weblogic.WLST $ORACLE_HOME/bin/soa-

createUDD.py --domain_home $DOMAIN_HOME --soacluster soacluster

2015 © Trivadis

SOA Cluster – Common Properties

The Audit Level property enables you to select the level of

information to be collected by the message tracking infrastructure

(None, Production, Development)

Don’t forget to turn off the Payload Validation at SOA Infra

Properties, if you don’t need validation at this level. This increases

the performance of the SOA Suite to greater extent. Major

Improvements are guaranteed.


2015 © Trivadis

SOA Cluster – Common Properties

More SOA Infra Advanced Configuration Properties... this link gives

you access to the soa-infra MBEAN

Configure the same value for all SOA Server

You can use WLST Scripts to make change


http://be-psd-03.trivadis.com:8020/em/faces/as/as/wlFarmHome?target=Farm_soa_domain&type=oracle_ias_farm&_afrLoop=7865989908019445

2015 © Trivadis

SOA Cluster – BPEL/BPMN Engine Level Properties

The BPEL thread pool

implementation notifies the

threads when a message has

been enqueued

It ensures the appropriate

number of threads are

instantiated in the pool

Verify Parameter

dspSystemThreads

dspInvokeThreads

dspEngineThreads

largedocumentthreshold


2015 © Trivadis

SOA Cluster – Mediator Engine Level Properties


Mediator is a component of

Oracle SOA that provides

mediation capabilities like

selective routing, transformation

and validation capabilities

Verify Parameter

metricsLevel

Parallel Worker Threads

(DeferredWorkerThreadCount)

Parellel Maximum Rows

Retrieved

(DeferredMaxRowsRetrieved)

Parallel Locker Thread Sleep

(DeferredLockerThreadSleep)

2015 © Trivadis


SOA Schema

2015 © Trivadis

SOA Schema – Database Purging (1)

The SOA schema has one of the biggest impact on the SOA business

application

Impact on performance of SQL query runtime

Impact on asynchronous routing rules and asynchronous BPEL

processes

It slows down the console access when querying for auditing data

Purging data needs database performance, do it in the general

maintenance window

If retention periods are defined too long, larger disk space is required –

BLOB column tables are growing fast


2015 © Trivadis

SOA Schema – Optimizing

Consider using hash partitioning on your tables and

indexes http://docs.oracle.com/cd/E29505_01/core.1111/e10108/others.htm#autoId10

Check for missing indexes High CPU Usage in SOA 11.1.1.5 due to Full Table Scan (Doc ID

1420114.1)

Don’t let your SOA Composite instance grow

exponentially

Define rules to keep the house clean, purge the

instances at regular interval to obtain better

performance from BPEL engine and Enterprise Manager


http://docs.oracle.com/cd/E29505_01/core.1111/e10108/others.htm#autoId10

2015 © Trivadis

SOA Schema – Purge Script (1)

Composite instances using the SOA Suite Service Engines

(BPEL, mediator, rules, BPM, etc.) are writing data to

tables residing within the SOAINFRA schema

Oracle provides a purge script to delete instances that are

no longer required using stored procedures that are

provided with Oracle SOA Suite out of the box

This purge script will delete composite instances that are

in the following states: Completed, Faulted, Terminated by user, Stale, Unknown

This purge script will NOT delete composite instances that

are in the following states: Running (in-flight), Suspended, Pending Recovery


2015 © Trivadis

SOA Schema – Purge Script (2)

There are two options for running the purge script:

Looped

Parallel

Basically the soa.delete_instance uses five parameters


Parameter Description

min_creation_date Beginning creation date for the composite instances.

max_creation_date Ending creation date for the composite instances.

batch_size Batch size used to loop the purge. The default value is

20000.

max_runtime Expiration at which the purge script exits the loop. The

default value is 60. This value is specified in minutes.

retention_period Retention period is only used by the BPEL process service

engine

2015 © Trivadis

SOA Schema – Package

Package soa.delete_instances


DECLARE

max_creation_date timestamp;

min_creation_date timestamp;

batch_size integer;

max_runtime integer;

retention_period timestamp;

BEGIN

min_creation_date := to_timestamp('2015-09-01','YYYY-MM-DD');

max_creation_date := to_timestamp('2015-09-30','YYYY-MM-DD');

max_runtime := 60;

retention_period := to_timestamp('2015-10-01','YYYY-MM-DD');

batch_size := 10000;

soa.delete_instances(

min_creation_date => min_creation_date,

max_creation_date => max_creation_date,

batch_size => batch_size,

max_runtime => max_runtime,

retention_period => retention_period);

END;

/

2015 © Trivadis


Testing

2015 © Trivadis

Testing - General

Performance optimization for a SOA Suite application is a real

challenge

A large number of components are involved in each request

Each component (JVM, DB, OSB, SOA, Network) has to be optimized

separately

To ensure that parameter changes have the desired effect, it is

necessary to repeat the same tests under the same conditions

several times

Maybe this change may have negative effects on other parts of the

application

Test cases must represent reality, use actual test data

The end scenarios is a representative mixture of all application

services


2015 © Trivadis

Testing

For each service, it is necessary to develop automatic test scenarios

which can be repeated for several times

For each scenario it is necessary to define

The expected response time

The number of requests that this service can be called per second

The expected response values

In reality, some query will be go wrong, it is important that this type of

queries are integrated in your test scenarios too (error handling)

During testing, don’t forget to check the log files…

Use SoapUI, LoadUI, Jmeter or whatever you want, each tool has his

own advantages


2015 © Trivadis

Testing – Verify your Test Runs

Start a test is one thing, optimize the system is another

For each component type, there are many specific tools to perform

these measurements

JVM -> jconsole, jvisualvm, jmc (Java Mission Control), DomainHealth

WLST, Enterprise Could Control, …

Domain -> DomainHealth, WLST, Enterprise Could Control, …

SOA -> DMS Spy, WLST, …

But only a few tools give you the possibility to see your complete

environment in the same application – E2E

EM12c Could Control, Dynatrace, AppDynamics


2015 © Trivadis

Testing – Dynatrace (1)

Real End-to-End Monitoring

A deep dive in the application code

Haben Sie Ihre WebLogic Umgebung im Griff

2015 © Trivadis


You can see in detail where your application spends time


2015 © Trivadis



The PurePath functions allows you to see the true transaction level

detail, all the time

2015 © Trivadis


See the database calls, code included

Search for the hostpot


2015 © Trivadis


Summay

2015 © Trivadis

Summary

Each components from a SOA Suite should be tuned separately

Do to this, you need automated test scenarios

To tune an application, you must understand how it works, speak with

your developers, eat with them, go to beer with them

To optimize the right parameters, you need to understand what

represents the measured value

Application Optimization is a permanent job - it must be started from the

beginning of the development


“This is as much an art as it is a science.” (John Mounjoy, WebLogic The Definitive Guide)

2015 © Trivadis

Fragen und Antworten...

2015 © Trivadis

BASEL BERN BRUGG GENF LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN

KOPENHAGEN

Daniel Joray

Principal Consultant

Tel. +41 58 459 50 26

[email protected]


Presentations & Public Speaking

Überleben im OSB/SOA Dschungel Daniel Joray