Tech

20121219

Diagnosable Oracle JDBC 11g Driver

The Oracle's 11g JDBC driver comes with a Self-Diagnosable features in which might help on the diagnose connection issue with the Data Base. Base on java.util.logging framework and javax.management MBean framework, in which follow the Java standard for monitoring and management.

  • java.util.logging _ Can be configure from providing a simple issue track, such as running out resource, to something more specific and detail tracing on an internal execution of a system. 
  • javax.management _ Can consult and change system's behavior through MBeans, accumulate statistics and notify in states changes on error's behaviors. 

The OJDBC self-diagnose, is not implemented in all the drivers. Actually you might need to go back to Oracles's driver site and download the following drivers:

  • ojdbc5_g.jar 
  • ojdbc6_g.jar

 *That's is why they have named these drivers with an *_g.jar at the end.

1- To enable the tracing you need to add at the java startup argument the following flag:

-Doracle.jdbc.Trace=true

This flags enable the global logging, if you want to debug the entire application this' the way to go.

2- Now we need to add the following flag on the java startup in which gives a path to an configuration file properties:

-Djava.util.logging.config.file=/jdbc/home/JDBCLogging.properties

3- Create the file /jdbc/home/JDBCLogging.properties; and set as following:

.level=SEVERE
oracle.jdbc.level=INFO
oracle.jdbc.handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=INFO
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter

This will throw the output into the console, there fore no log file should be created. If you need to actually to produce a physical log you need to set as following:

.level=SEVERE
oracle.jdbc.level=INFO
oracle.jdbc.handlers=java.util.logging.FileHandler
java.util.logging.FileHandler.level=INFO
java.util.logging.FileHandler.pattern=jdbc.log
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter


The amount of details are controlled by choosing bellow values, from less information to most detail information:

  • OFF
  • SEVERE
  • WARNING
  • INFO
  • CONFIG
  • FINE
  • FINER
  • FINEST
  • ALL
The 11g's OJDBC also lets you choose which system or part of the driver in which can be debugged; Is a nice thing, because some times you do not need too much information but specific information, and how to do this is controlling the logging level using the following flags:

  • oracle.jdbc 
  • oracle.jdbc.driver
  • oracle.jdbc.pool
  • oracle.jdbc.rowset
  • oracle.jdbc.xa
  • oracle.sql
*Subject to change by release version.
These flags can be simply add to the  /jdbc/home/JDBCLogging.properties like:

.level=SEVERE
oracle.jdbc.level=INFO
oracle.jdbc.driver.level=ALL *
oracle.jdbc.handlers=java.util.logging.FileHandler
java.util.logging.FileHandler.level=INFO
java.util.logging.FileHandler.pattern=jdbc.log
java.util.logging.FileHandler.count=1
java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter


ref: 








20121214

java.lang.OutOfMemoryError: GC overhead limit exceeded

The OOM: "GC overhead limit exceeded", only means that Full GC is doing much work for nothing. Too much time is being spent on Garbage collection while the collection it self recover less than 2% of the heap.

Accordingly this OOM can only occurs in two different GC algorithm, such as:

  • Parallel Collection: Parallel's automatic tuning, provide a maximum pause time. 
  • Concurrent Collection: The concurrent collector pauses an application twice during a concurrent collection cycle.

Only collections in which application is stopped, and is due to concurrency mode failure or explicit collection request. This OOM is to prevent applications to be stopped for a long period of time while collecting on a small heap.


Possible Solutions:

  • Use another GC algorithm;
  • Check if your heap is too small to run Parallel or Concurrent GC. If this is the case tune your heap size accordingly;
  • Turn off this feature by using: -XX:-UseGCOverheadLimit. This flag replaced the  UseGCTimeLimit, in which is referenced on the CR 6287811; 
  • Try use the -XX:MaxGCPauseMillis=n, in which the GC will try its best to achieve this value. No promises made on this one. 

Bug:
      The CR 6287811, is about the log output error in which mislead many people in thinking there were an issue with the heap:

  • before: "java.lang.OutOfMemoryError: Java heap space";

  • now: "java.lang.OutOfMemoryError: GC overhead limit exceeded";


Doc:

-XX:+UseGCOverheadLimitUse a policy that limits the proportion of the VM's time that is spent in GC before an OutOfMemory error is thrown. (Introduced in 6.)



ref:
  • http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
  • http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
  • http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6287811

20121213

Using Wildcard asterisk (*) SSL with WLS

Setting the certificates to use the wild card asterisk (*) can be troublesome on the WLS.

Example of wildcard use:

cn = *.example.com

Since WLS 10.3.6 and 12.1.+ has been hard-coded a new host name verifier on the WLS in which with some simple configuration you can use your wildcarded certificates:

A_ Start WLS with the following java properties:

./startWebLogic.sh -Dweblogic.security.SSL.HostnameVerifier=weblogic.security.utils.SSLWLSWildcardHostnameVerifier

or

B_

1. Start the AdminConsole and goto selected Server::Configuration::SSL;

2. Click on the link Advanced;

3. Copy/paste the weblogic.security.utils.SSLWLSWildcardHostnameVerifier Class name on the Hostname Verifier field;

4. Save;

C_ Restart server and test your SSL connection.


*For previous version of WLS 10.3.x there's a patch in which, I recommend to open an SR with Oracle to retrieved.


Links:
Using the Wildcarded Host Name Verifier
Configure a custom host name verifier
Oracle WebLogic Server API Reference SSLWLSWildcardHostnameVerifier

20121004

Testing JavaScript on Blogger

Nice to know that I can play a bit with javaScript on blogger. Even if I have to do things by hand... but that is cool...

My example in action is bellow:

testing Script

will be date here.

20121003

java.lang.InternalError: Unexpected CryptoAPI failure generating seed


Throwable: java.lang.InternalError: Unexpected CryptoAPI failure generating seed
Stack Trace:
java.lang.InternalError: Unexpected CryptoAPI failure generating seed
at sun.security.provider.NativeSeedGenerator.getSeedBytes(NativeSeedGenerator.java:43)
at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
at java.security.SecureRandom.generateSeed(SecureRandom.java:495)

While generating seed to create random numbers, the exception is being thrown form the method bellow:

{
        // fill array as a side effect
        if (nativeGenerateSeed(result) == false) {
            // should never happen if constructor check succeeds
            throw new InternalError
                            ("Unexpected CryptoAPI failure generating seed");
        }
}

ref:  sun.security.provider.NativeSeedGenerator

To generate random numbers, SSL security code relies upon entropy on a machine. Entropy is activity of the machine,If entropy is minimal or non-existent, then the random number generator will be slow and security operations may time out.

The class generates seeds for strong cryptographically number generator. It uses two techniques:

Computing current system activity:
Default, produced by counting the number of times the VM manages to loop in a perioud of time. Does not reflect machine load, and a number of sleeper threads are generated to add entropy.
Entropy gathering device:
Alternative, is to acquire material from entropy gathering device, such as /dev/random. By setting the "securerandom.source" security property of the /lib/security/java.security file.
Use the bellow command that starts the Java process as a possible solution:
  • -Djava.security.egd=file:///dev/urandom
  • or
  • -Djava.security.egd=file:/dev/./urandom
For further information, see Sun bugs 6202721 and 6521844 at:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6202721
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521844

Using the WebLogic JarBuilder to power java client programs.

This post is a reference in how to export a full jar with all WLS libs. One good reason is to transport WLS proprietary classes into a java client application. That way a full use of WLS's libs.

Usually engineers only copy/paste the weblogic.jar, but the right way to do it, is to use the WebLogic JarBuilder tool, bellow will show you the following:

  • Creating a wlfullclient.jar for JDK 1.6 and 1.7 client applications
  • Creating a wlfullclient5.jar for JDK 1.5 client applications
wlfullclient.jar file since WLS 10.3
  1. goto: $cd WL_HOME/server/lib
  2. run: $java -jar wljarbuilder.jar
  3. On the same directory you should see this wlfullclient.jar
  4. Copy over to client application's classpath
wlfullclient.jar file for WLS 10.0
  1. goto: $cd WL_HOME/server/lib
  2. run: $java -jar wljarbuilder.jar -profile wlfullclient5
  3. On the same directory you should see this wlfullclient5.jar
  4. Copy over to client application's classpath

The WL_HOME reference to wlserver_12.1, as an example. Also keep in mind the WLS start using JDK 1.6 since WLS 10.3

ref:Using the WebLogic JarBuilder Tool

Starting Coherence with WLS Console.

Easy steps... 
  1. Goto: Admin Console::Servers::Coherence Servers::myCoherenceServer::Configuration::Server Start
  2. Add these two jars to the classpath
    • $MW_HOME/modules/features/weblogic.server.modules.coherence.server_12.1.1.0.jar:$MW_HOME/coherence_3.7/lib/coherence.jar
    • Replace $MW_HOME by the actual directory path of the Middleware Home
  3. Start the Coherence Server from the WLS Admin Console to verify it works fine now. 

Notice, this example is for WebLogic Server 12.1.1

20121001

High Availability for WLS, How to Quickly pack/unpack copy the domain for testing.

We have a domain in which is in production, and you quickly need a testing domain with the same configuration. Since the production domain is in production, you cannot actually run the risk of messing things up. Therefore I am quickly writing this so it helps you on getting a copy of your domain quickly running for testing;

Navegate to your wls home:

1. [Middleware]$ cd 1036bin/wlserver_10.3/common/bin/

2. [bin]$ ls
commEnv.sh  config_builder.sh  config.sh  pack.sh  setPatchEnv.sh  startDerby.sh  startManagedWebLogic.sh  stopDerby.sh  unpack.sh  upgrade.sh  wlscontrol.sh  wlsifconfig.sh  wlst.sh

Use the pack command to build a template, here I am using the tmp directory:

3. [bin]$ ./pack.sh -domain=/home/Oracle/Domains/1036bin -template=/tmp/mydomain.jar -template_name="testingBackup"
<< read domain from "/home/Oracle/Domains/1036bin"
>>  succeed: read domain from "/home/Oracle/Domains/1036bin"
<< write template to "/tmp/mydomain.jar"
....................................................................................................
>>  succeed: write template to "/tmp/mydomain.jar"
<< close template
>>  succeed: close template

Unpack on a different directory, Actually the newDirectory does not exist the unpack will create one:

4. [bin]$ ./unpack.sh -template=/tmp/mydomain.jar -domain=/home/Oracle/newDirectory
<< read template from "/tmp/mydomain.jar"
>>  succeed: read template from "/tmp/mydomain.jar"
<< write Domain to "/home/Oracle/newDirectory"
...............................................................................................
>>  succeed: write Domain to "/home/Oracle/newDirectory"
<< close template
>>  succeed: close template

Goto the newly created newDirctory and start your backup domain:

5. [ bin]$ cd /home/Oracle/newDirectory

6. [newDirectory]$ ls
autodeploy  bin  config  console-ext  fileRealm.properties  fileRealm.properties.bak  init-info  lib  pending  security  servers  startManagedWebLogic_readme.txt  startWebLogic.sh  tmp

6. [ newDirectory]$ ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms512m -Xmx512m
.
WLS Start Mode=Development
.
.
.
<Oct 1, 2012 2:49:47 PM CLT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING>
<Oct 1, 2012 2:49:47 PM CLT> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

This example was run into the same Linux machine, but if you want to transport this domains to a different machine you may just copy the mydomain.jar into the new machine, install the same WLS version and run the steps from 4 - 6 (While WLS installation you may choose the mydomian.jar as a template). This action is mainly use for extending domain into a remote machine, but also can be use to build testing environments with the same configuration as the production.

20120924

How to set and Start Coherence with WLST




Oracle Weblogic Server - Version 10.3.4 to 12.1.1.0
Oracle Coherence - Version 3.6.0 to 3.7.1.x

Different ways to start Coherence Servers using WLST?

Pre-requirements:

Node Manager should be running and please make sure that you have the following jar files on your classpath;

/home/Oracle/Middleware/121cJar/modules/features/weblogic.server.modules.coherence.server_12.1.1.0.jar:/home/Oracle/Middleware/121cJar/coherence-java-3.7.1.0b27797/coherence/lib/coherence.jar

Use the following links to create and set the Coherence Servers using the Admin Console:

http://docs.oracle.com/cd/E25054_01/apirefs.1111/e13952/taskhelp/coherence/CreateCoherenceServers.html

http://docs.oracle.com/cd/E25054_01/apirefs.1111/e13952/taskhelp/coherence/ConfigureStartupArgumentsForCoherenceServers.html


Ans1: Using the nmStart():

1. Start the WLST tool by setting the environmnet from Domain/bin directory:

$. bin/setDomainEnv.sh (cmd if Windows)
$java weblogic.WLST

2. Connect With the NM;

wls:/offline> nmConnect("weblogic","Welcome1","localhost","5556","12cjar","/home/Oracle/Domains/12cjar")

3. Start the Coherence server in which is named here as Server-0;

wls:/nm/12cjar> nmStart("Server-0",serverType="Coherence")

4. The Coherence server should be started:

Starting server Server-0 ...
Successfully started server Server-0 ...



Ans2: Now let use the CoherenceServerLifeCycleRuntimeMBean to start:

1. Set the domain environment from your WLS domain and run the following:

$java weblogic.WLST

2. Connect with your running AdminServer.

connect()

3. goto your domain Runtime.

domainRuntime()

4. goto your Coherence MBeam home.

cd("CoherenceServerLifeCycleRuntimes/Server-0")

5. and run the server using this command:

cmo.start()

The Server should be running by now....



Ans3: Goto the link bellow and you should see the startCoh.py and stopCoh.py, then is the matter of using the following command:

1. Create the files stopCoh.py and startCoh.py;

2. Goto bin directory of your domain.

3. Set your domian environments: $ . setDomainEnv.sh

4. Go back to the directory in which you created the two py files.

5. and run for example:

$java weblogic.WLST startCoh.py

ref: http://docs.oracle.com/cd/E24329_01/web.1211/e24442/coh_wls.htm#CIHDDAAE



Ans4: The Most common way is to use the Console, for that you can use Oracle's documentation:

http://docs.oracle.com/cd/E25054_01/apirefs.1111/e13952/taskhelp/coherence/StartCoherenceServers.html

20120810

Node Manager Starting Weblogic Server life cycle explained


Node Manager, is an java instance that ensure high availability of WLS servers with whole server migration and  automatic restarting WLS nodes in case of unwanted shutdown. NM uses the ServerMBean, ServerStartMBean, and SSLMBean to set the JAVA_VENDOR, JAVA_HOME, JAVA_OPTIONS, SECURITY_POLICY, CLASSPATH, and ADMIN_URL. Therefore by using the NM to start a managed server does not necessary uses the domain's bin directory startWebLogic.sh or startWebLogic.cmd to start a node.

This is due because the flag StartScriptEnabled nodemanager.properties file is set to false by default; So this means that all the settings modification done, such as adding debug flags or adding a new jar on the classpath, on startWeblogic.sh will not be pick up or better saying will not be provided to the JVM.

So solve this unpleasant situation you can just go to wlserver_10.3/common/nodemanager directory and edit the nodemanager.properties as following:

StartScriptEnabled=true

Also, you need to pay attention on another configuration on the same file; The StartScriptName actually tells the NM which script should be use to start the WLS node. By default it uses the startWebLogic.sh, but you can create a new file in the domain's bin directory with all needed start-up configurations;

StartScriptName=startWebLogicModified.sh

Understand the some situations you are using the NM to start the WLS nodes without knowing; Sometimes this process seems to be unseeing by administrators. Bellow is the list of situations in which you may be using NM to start servers:

Admin Console - You need to have the NM running to start any instance.

WLST -
1. When you use the following WLST's commands:
start NM:
offline> startNodeManager()
Connect: offline>nmConnect('username','password','nmHost','nmPort','domainName','domainDir','nmType') 
Start Serverwls:/nm/mydomain>nmStart('AdminServer')
Stop Server:
wls:/nm/mydomain>nmKill('serverName')


2. When you use the WLST to connect to the AdminServer, therefore is just like when you use the Admin Console:
Connecting with AdminServer:
(offline)> connect('username','password')
Then you can start by using single server name or cluster:
wls:/mydomain/serverConfig>start('managedServerName','Server')
wls:/mydomain/serverConfig>start('clusterName', 'Cluster')

Pay attention that some basic rules must be follow, for NM to start a server:

1. Any particular server mus be assign to the NM; Do not go around setting partially the cluster to start with NM and hope that all the servers will start. 

2. Is one NM per physical machine; WLS has the great feature of extend a domain across physicals machines, so I insist again remember to assign the WLS nodes to the particular NM located on the same physical machine. 

3. Remember that NM by default uses the ServerStartMBean to set the environment, so if you have edited the startWebLogic.sh the JVM will not pick up the changes. Unless, as explained, you modify these
StartScriptEnabled and StartScriptName accordingly with your requirements on the nodemanager.properties file.

20120809

How to Install OpenJDK on Ubuntu

A. To install the OpenJDK on ubuntu is very easy and one shot deal. First you need to search for the JDK version. First open a terminal and run the following command:

$ sudo apt-cache search openjdk-

This should give you some choices like:


openjdk-6-doc - OpenJDK Development Kit (JDK) documentation
openjdk-6-jdk - OpenJDK Development Kit (JDK)
openjdk-6-jre - OpenJDK Java runtime, using Hotspot JIT
and 
openjdk-7-doc - OpenJDK Development Kit (JDK) documentation
openjdk-7-jdk - OpenJDK Development Kit (JDK)
openjdk-7-jre - OpenJDK Java runtime, using Hotspot JIT

many others packages should appear. 

OR

you can just type in the following command: 

$ sudo apt-get install openjdk- 

and tab twice!

B. To install the OpenJDK now you can run the following command and similar to the one above: 

$ sudo apt-get install openjdk-6-jdk

Just follow the screen instructions and that's it. 

C. In case you installed both OpenJDK-6 and OpenJDK-7, you can simple switch between both by using this command. 

$ sudo update-alternatives --config java

Then you should see this: 
There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                           Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java   1061      auto mode
  1            /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java   1061      manual mode
* 2            /usr/lib/jvm/java-7-openjdk-i386/jre/bin/java   1051      manual mode

Press enter to keep the current choice[*], or type selection number: 

I have selected 2 to set the OpenJDK-7 as the default java. 

D. In case you want to set any program java home to the non-default, in my case OpenJDK-6. You can just provide the path: /usr/lib/jvm/java-6-openjdk-i386 




What about Service Guardian on Coherence

Service Guardian is basically a stuck thread watch dog for Coherence cluster, which consist in sent heartbeats from owned created  Coherence's thread; In case a thread from a specific node fails to respond the heartbeat some time-out flags are triggers, for corrective action to be taken.

The time-out recoveries works like:

Soft time-out _ Coh. attempts to interrupt the thread before the Hard time-out is reached. If successful normal Processing resumes.


<Error> (thread=DistributedCache, member=1): Attempting recovery (due to soft
timeout) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper(com.
tangosol.examples.rwbm.TimeoutTest),5,WriteBehindThread:CacheStoreWrapper(com.
tangosol.examples.rwbm.TimeoutTest)]", State=Running}


Possible some network delay or latency.305000 milliseconds is the default value, and there is no action required, unless you frequently see this log output. Which means that you might need watch your network traffic and do some tuning. Also you may change the default value to better fit your necessities.

Hard Time-out _ after the set timing, this case the default 305000 milliseconds is reached Coh. now tries to stop the thread. 


<Error> (thread=DistributedCache, member=1): Terminating guarded execution (due 
to hard timeout) of Daemon{Thread="Thread[WriteBehindThread:CacheStoreWrapper
(com.tangosol.examples.rwbm.TimeoutTest),5,WriteBehindThread:CacheStoreWrapper
(com.tangosol.examples.rwbm.TimeoutTest)]", State=Running}

The Coh. thread is not behaving as expected, possible doing some investigation by thread dumps might help identify the issue. But first you need to identify which node in which should take thread dumps. From the log above it gives you hints like "thread=DistributedCache, member=1", the thread is DetributedCache and the member is 1. 

305000 milliseconds, if I'm not mistaking should be about 5 minutes. There fore running about 15 thread dumps, each 30 seconds should help analyse, in this case why the DistributedCache is taking too long. Do not disregard network traffic, some issues can be resolve by using Coh.'s Unicast and Coh. WKA.

Settings for Unicast:


-Dtangosol.coherence.localhost=192.168.0.1
-Dtangosol.coherence.localport=8090
-Dtangosol.coherence.localport.adjust=true


Settings for Well Known Addresses:

-Dtangosol.coherence.wka=192.168.0.100
-Dtangosol.coherence.wka.port=8088



Lastly _ The dead end, after all fails you are done for it... Naahhhh!!! At this point Coh. actually tries to follow policies like:

  • Shutting down the cluster service: 
The faulty node stop all its cluster communication on an attempt to reset all the distribution services. Depending on your logging level and size of cluster, this could be a pain.  
  • Shutting down the JVM:
I am not really experience with this behaviour, but one thing is for sure, we would know which node is the cupid; I understand that WLS's Node Manager can start Coh. cache servers, and also that Node Manager can restart WLS servers... hummm... But I am not sure if the Node Manager, but any ways bellow is some interesting links. 

Start Coh. Servers from the WLS's Admin Console:
http://docs.oracle.com/cd/E28271_01/apirefs.1111/e13952/taskhelp/coherence/StartCoherenceServers.html
How NM restart Managed Servers:
http://docs.oracle.com/cd/E23943_01/web.1111/e13740/overview.htm#i1074986
 
  • Performing a custom action:


This option means that you have known situations in which Coherence threads might take longer than expected or would like to have more control on this feature by controlling its behaviour. But is preferable that you follow the Coh.'s  documentation for this settings.
ref: http://docs.oracle.com/cd/E24290_01/coh.371/e22837/api_guardian.htm

Just known that Service Guardian is a new feature on Coherence, which was introduce on the 3.5 version. This service is reaching some good maturity on Coh. 3.7.1.xx, therefore upgrading to the latest version of Coherence is a must to avoid defects. One last thing, in case you just do not want to go so deep into this feature you can always disable or even raise the time-out value:

Shut-down Guardian:
-Dtangosol.coherence.guard.timeout=0
Raise time-out in milliseconds:
-Dtangosol.coherence.guard.timeout=700000
Hard coding time-out:
import
com.tangosol.net.GuardSupport
set heartbeat
GuardSupport.heartbeat();
known long running operation
GuardSupport.heartbeat(long cMillis);











20120806

Weblogic Server Thread pool, from my view.


Weblogic Server has two different way of handling Thread pools: 

Weblogic 8.1 Thread Pool Model

The process was performed in multiple execute queues. Different types of work were handle in different queue, basing on priority and ordering requirements to avoid dead lock. The tuning on this model is very manual and dependent on the administrator analyses and configuration to get some real performance. The control of thread was basically changing the number of threads in the default queue, or configure custom execute queues to a particular applications to have access to a fixed number of execute thread. Oracle /BEA recommends migrating to Work Manager. 

How to enable

- The configuration to use the 8.1 is not trivial as configuring from console; As mentioned above the recommendation is to use the Work Manager, since this style is very prone to human bad tune configuration. I mean, system resources changes constantly just by adding  a new program or changing parameters, maybe a good tune on a system and a small change happens you could face some performance degradation. 

1. Shut down the WLS's java instance. 
2. Edit the config.xml, by adding the use use81-style-execute-queues element to true. 
3. Start a new WLS's java instance. 
4. Explicitly create the weblogic.kernel.Default execute queue from the Console.
5. Reboot the WLS's server java instance. 

<server>
   <name>YourServer
n
</name>
   <ssl>
      <name>myserver</name>
      <enabled>true</enabled>
      <listen-port>7002</listen-port>
   </ssl>
   <use81-style-execute-queues>true</use81-style-execute-queues>
   <listen-address/>
</server>

*You might need to do the same steps for each server. 

Tuning 

The ThreadCount of the element ExecuteQueue set in the config.xml equals the number of simultaneous operations that can be performed by an application assign to the execute queue. Threads consume resources, and having too many as consequence can have a lot physical resources unnecessary use for unnecessary work. Therefore could decrease performance of the application as for the entire system. 

The ThreadCount can be different depending in what type of Mode that you start the WLS: 

Development: Mainly use to load test and development environment, this could mean that WLS can have more I/Os and other types of issues while using this mode. This mode is mainly to stress the application container it self and the default number of thread is 15

Production: As named this mode should be use for production. Changes are not on the fly, and usually the application server needs some extra steps, lock edit is one of the example; This mode is for performance, and passing on the stress to the environment system and the default value of thread is 25

*Some cases, depending on large cluster and configuration, just by changing the mode on the WLS from Development to Production you can face some performance degradation on your environment. Simple math, with default 8.1 configuration: 
10 instances of Development mode, equals to 150 threads to share the system resources. 
10 instances of Production mode, equals to 250 threads to share the system resources. 

Simple tuning of thread could resolve this issue; I would disregard having the Porduction mode being remove from an production environment. Check some scenarios for Modifying the default thread count: 

(You need to understand the nature of your business, some applications has a pick time of usage at the end of every month; example: banking, after you getting your salary).

Thread Count < numbers of CPUs :
  • Behavior _  CPUs are not being fully use on a pick time, but there is some work to be done.
  • Possible Action _ Increase the thread count. 

Thread Count == number of CPUs: 
  • Behavior _ CPUs are not being fully use on a pick time, but there is some work to be done.
  • Possible Action _ Increase the thread count. 

Thread Count > number of CPUs (by a moderate number of threads)
  • Behavior _  CPUs are being fully use on a pick time, with a moderate amount of context switching.
  • Possible Action _  Tune the thread count, and test performance. 

Thread Count > number of CPUs (by a large number of threads)
  • Behavior _  CPUs are being fully use on a pick time, with many context switching.
  • Possible Action _ Reduce the number of threads. (remember my example of Modes above...)



Work Manager

Is a single thread pool, where all types of work are executed. The tuning of an Work Manager is by demand, in which means that it happens automatically. Queue monitors the throughput base on history and determines the adjust of thread count. The work is base on defined rules, runtime metrics (history) to avoid deadlocks. This the WLS passes it on some of  the tuning responsibility from the administrators to it self. 

Administrator can manage work by configuring some types of scheduling guideline by defining some components: 

  • Fair Share Request Class
    Specifies the average time required to process a request. Default is 50. (This is base on percentage, therefore timing also will depend on the environment physical capacity to process).
  • Response Time Request Class
    Is the response time goal in milliseconds. Not applied to individual request. 
  • Min Threads Constrains
    The guarantee  number of threads the server will allocate. Default is Zero. 
  • Max Threads Constraint
    Limit the number of concurrent threads. Default is unlimited, -1. 
  • Capacity Constraint
    Forces the server to reject request when reached its capacity, either individual or global capacity is exceeded it will reject requests. The default is unlimited, -1. 
  • Context Request Class
   Assigns request classes depending on context information. 
WLS works as best effort, this means will no guarantee that configured ratio will be maintained. Its behavior can possible change depending on demand.

ref: http://docs.oracle.com/cd/E21764_01/web.1111/e13814/appb_queues.htm
ref: http://docs.oracle.com/cd/E24329_01/web.1211/e24432/self_tuned.htm#i1068066

Important: This information is based on my personal understanding of Oracle's documentation. Please refer to Oracle's documentation for further information and tuning. Remember that you as an administrator is responsible for the best tuning of your own environment. Simple word, use your gut instincts! if you have some problem with this please read the article written by Leon Watson from MailOnline:
"Researches say our first thought is often our best" 
ref: http://www.dailymail.co.uk/news/article-2031848/Why-right-trust-gut-instincts-Scientists-discover-decision-IS-right-one.html

20120802

Weblogic's console slowness performance issue.


I have notice that after installing the latest WLS with the latest JDK 1.7 on a redhat base 64 bit linux, the Admin Console is taking a long time to show on browser. The Admin server it self starts fine, but when I provide the username and password then I would have to wait for about 3 - 4 minutes to have access.

Here is my system config:

WLS 12.1.1
JDK 1.7.0_5
redhat linux based x86_64

Since this strange behavior actually happens during authentication and authorization of my user/password, this must be related with some security performance issue. From my past experience,  I learned that the linux OS's  /dev/random and /dev/urandom should have some effect on this behavior. Please, check wikipedia for further knowledge on ramdom and urandom.

whit some googling, I came across many links in which talks about starting performance by adding the following command while starting the AdminServer:

$./startWeblogic.sh -Djava.security.egd=file:/dev/./urandom

This did the trick, and then I found a open java bug 6202721 in which was closed as "not a java bug"... 


The other definitive way to change on all the servers starting on the same JDK  you can just edit the java.security file: 

#cat /usr/java/jdk1.7.0_05/jre/lib/security/java.security|grep "urandom"
# On Solaris and Linux systems, if file:/dev/urandom is specified and it
# This "NativePRNG" reads random bytes directly from /dev/urandom.
# On Windows systems, the URLs file:/dev/random and file:/dev/urandom
securerandom.source=file:/dev/urandom
#   -Djava.security.egd=file:/dev/urandom

vi the java.security file and change
from: 
securerandom.source=file:/dev/urandom

to: 
securerandom.source=file:/dev/./urandom

hopefully works for you.