SOA Suite Knowledge – Polyglot Service Implementation with Groovy

Polyglot programming, defined most simply, is the use of multiple languages in software development. Implementing services on the SCA Container is already a intended polyglot development approach. Oracle SOA Suite have the SCA service implementation types of BPEL, Human Workflow, Mediator, Rule and Spring Components. These components are mixing the General-Purpose Language (GPL) Java with Domain-Specific Languages (DSL) like XSD, WSDL or Schematron. But Spring Components are also enabler of service implementations with other JVM GPL languages beside Java. 

In my opinion Neal Ford was absolutely right when he coined the term “Polyglot Programming” and predicts “Applications of the future will take advantage of the polyglot nature of the language world … The times of writing an application in a single general purpose language is over” already in 2006. In order to produce polyglot implementations you need environments where polyglot-ism is required or at least encouraged. Oracle SOA Suite is such a polyglot supporting environment and you are doing polyglot development all the time (e.g. XML, WSDL, SQL, Java). But also on the GPL side the developers or not limited on Java. SCA Spring Components are supported since Patch Set 2 and the Spring Framework supports dynamic languages since version 2 (BeanShell, JRuby and Groovy).

Groovy is a General-Purpose Dynamic Language and has the best and seamless integration with Java (beside Spring integration Groovy supports also direct and JSR 223 integration). Therefore, using Groovy has a low threshold for Java developers; it is easy because Groovy has a Java-like syntax. Most Java code is also syntactically valid Groovy. So it’s an iterative learning approach for Java developers switching to Groovy. My first contact with Groovy was at the JAX conference in 2008. It was a Groovy newbie session from Dierk König (well-known book author of "Groovy in Action"). And for the first time in a long while of Java programming I had an awakening and still addicting interest on this development language.

The Groovy language is meant to complement Java and is adding a wide range of features that are sadly lacking in Java (for example Closures, Dynamic Typing and Metaprogramming – just to name a few). Concurrency and parallelism is increasingly important in the era of multi-core CPUs with a growing number of CPU cores. GPars is part of Groovy and offers a intuitive and safe way to handle tasks concurrently. Groovy makes it also easy to create DSLs (to simplify the “solution”). Optional parentheses, optional semicolons, method pointers and Metaprogramming let the code viewed as “executable pseudo code” (and easy readable by non-programmers). One of my personal favorites on using Groovy is the easy XML handling, both for consuming and producing XML content with its XML builders and parsers. Therefore I encourage you to take a deeper look on Groovy.

In my opinion Groovy envision also a bright future because the Spring development team announced to put a strong focus on Groovy on the upcoming Spring Framework 4.0. Spring 4 will about properly supporting Groovy 2 as a first-class implementation language for Spring-based application architectures - not just for selected scripted beans but rather for an entire application codebase, as a direct alternative to the standard Java language (quoting on Jürgen Höller from SpringSource who is the Spring Framework co-founder and project leader). The Spring Framework is the most popular application development framework for enterprise Java and will become therefore the driving force for getting more Java developers in touch with Groovy (and let them feel what is the “right language for the job”). 

The latest version of Oracle SOA Suite (v11.1.1.7) comes with Spring v2.5.6 and has also the Groovy v1.6.3. libraries on board. These versions are outdated, because Spring v.2.5.6 was released in November 2008 and Spring is already on the 3.x version. Groovy v1.6.3 was released in Mai 2009 and Groovy is today on version 2.x. Anyway, since Oracle SOA Suite PS5 (11.1.1.6) it’s possible to do Spring Bean implementation using Groovy. Oracle itself is also using Groovy, for example the Rule Reporter is written in Groovy or Oracle ADF is using Groovy as well. But the official documentation on how to write SOA Suite Spring Beans with Groovy should be improved, because you need more details to make our polyglot implementation running. Motivating for using Groovy and showing the details are the reasons for this blog. Now lets go to the details.     

First you have to do some configure steps. I show you the steps with the out-of-the-box coming Groovy library v1.6.3. But you also could download the latest Groovy version and make use of the newest Groovy features (I did a successful test with Groovy v2.1.3).

1.) Copy the $MW_HOME/oracle_common/modules/groovy-all-1.6.3.jar library to the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 folder

You can add JAR files and custom classes to a SOA composite application. A SOA extension library for adding extension JARSs and classes to a SOA composite application is available in the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 directory. The Groovy library has become known to SOA composite applications.

2.) Run ANT at the $ORACLE_HOME/soa/modules/oracle.soa.ext_11.1.1 folder

Running ANT on the "soa.ext" folder will update the oracle.soa.ext.jar library with a changed MANIFEST.MF file, adding a reference to Groovy.

Warning: This procedure is not cluster-friendly. If you're running a cluster, you must copy the file and run ANT on each physical server that forms the domain.

3.) Add the Groovy library to the Weblogic system classpath. Open the setDomainEnv configuration file and put the Groovy library on the POST_CLASSPATH setting. 

4.) Restart the Weblogic application server

Once the configuration is done, the Spring Bean Groovy coding could start. I’m reusing the Calculator example from an older blog about Spring Bean Contract-First. Therefore I copy the Calculator WSDL on the project folder, create a Web Service Adapter and pointing on the Calculator WSDL file. Afterwards I create an Spring Bean component and wire the Spring Bean with the Web Service Adapter (contract-first approach). The JDeveloper wizard creates the necessary Java files for the Spring Bean.

I’m doing only one change on the setter methods of the generated type classes for the response (AddResponseType, SubtractResponseType, MultiplyResponseType and DivideResponseType). The setter should return “this”. This approach is commonly called a builder pattern or a fluent interface. You don’t need to care about the returned object if you don't want, so it doesn't change the "normal" setter usage syntax. But this approach allows you also to return the changed class (I’m using this approach together with the end return Groovy feature) and to chain setters together.

Next step is the Groovy implementation of the generated Java ICalculatorService interface. I’m placing the GroovyCalculator class on the same package name like the Java interface but on the <project>\SCA-INF\src folder instead of the <project>\src folder (details below).

The implementation is coming in Groovy style. Semicolons and the default public visibility declaration disappeared. Variables are dynamically typed (keyword def). The mathematical operations are Closures (support for Closures in Java is planned for Java 8; recently postponed to March 2014). The last expression result will be returned (so-called end return), therefore the keyword return is in Groovy optional. In general, Groovy code is less verbose, and more expressive and compact.

You have to make sure that the Groovy class is coming on the SCA-INF/classes folder at the SOA archive (SAR) JAR file. Therefore you have to place the Groovy class file on the SCA-INF/src sub-folder and instruct JDeveloper to copy also files with groovy extension on the Output Directory (Project Properties –> Compiler –> Copy File Types to Output Directory).

The final step involves defining dynamic-language-backed bean definition on the Spring Bean Context file (the wiring-XML-file). One for each bean that you want to configure (this is no different to normal Java bean configuration). However, instead of specifying the fully qualified classname of the class that is to be instantiated and configured by the container, you use the <lang:language/> element to define the dynamic language-backed bean. For Groovy you have to use the <lang:groovy/> element (which instructs Spring to use the specific Groovy-Factory).

Afterwards the SCA Composite could be deployed and tested. The Groovy script will be automatically compiled during the deployment on the application server. It is worth to mention that Spring supports also refreshable beans and inline scripting.

Refreshable beans allow code changes without the need to shut down a running service or redeploy the service. The dynamic-language-backed bean so amended will pick up the new state and logic from the changed dynamic language source file. Be careful, in my opinion it’s a powerful but dangerous feature. One reason for inline scripting could be a quick Mock Bean implementation (take a look on the Mock example below).

The Mock implementation is always returning 42 as a result.

Groovy coding errors will result in a ScriptCompilationException giving you more details about the occurred issue (reason, line, column). For example …

There was an error deploying the composite on SOAServer: Error occurred during deployment of component: CalculatorSB to service engine: implementation.spring, for composite: Calculator05Groovy: SCA Engine deployment failure.: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'GroovyCalculatorBean': BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'GroovyCalculatorBean': Could not determine scripted object type for GroovyScriptFactory: script source locator [classpath:calculator/example/suchier/GroovyCalculator.groovy]; nested exception is org.springframework.scripting.ScriptCompilationException: Could not compile script; nested exception is org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed, GroovyCalculator: 17: unexpected token: this @ line 17, column 84.

Often I’m using SoapUI beside Enterprise Manager Fusion Control for doing first tests. Here is the test SoapUI output for a valid division.

This is the output for a division by zero. The exception is throw on the Groovy class, but is also a standard based SOAP fault.

The source code is available as 7-zip here.

That’s it. This is not a bash-the-other-languages blog. But my personal favorite language beside Java is Groovy and there many reasons to take a look on Groovy also as a SOA Suite developer. Now you have the details to start Groovy based polyglot service implementation on Oracle SOA Suite. If you are a Groovy beginner, I would recommend to take a look at the excellent books “Groovy in Action” (from Dierk Koenig) and “Groovy Recipes” (from Scott Davis).

SOA Suite Knowledge – MTOM enabled Web Services

You might have heard about MTOM/XOP. Possibly you have a broad idea of what MTOM/XOP is doing. Than you start from the same initial situation like me some months ago. Let me give you a jump start on what MTOM/XOP means in general and in particular on the Oracle platform.

MTOM is the W3C standardized Message Transmission Optimization Mechanism which is used in combination with the concrete referencing mechanism XOP which stands for XML-binary Optimized Packaging. MTOM/XOP is a way to handle opaque data with XML like binary data of images, audio or video. MTOM/XOP does not use any compression and decompression technique. There is another standard called Fast Infoset that addresses the compression issue. But large application data which need to be compressed for efficacy could be also handled by MTOM/XOP (more about it on the given example below).

XML parsers will fail when you place binary data in a text node of a XML document. The binary data could contain reserved XML characters like ‘<’, ‘>’ or ‘&’ which will break the parser on the “other side”. CDATA wrapping allows these reserved XML characters but the parser is looking for a ‘]]>’ (which marks a CDATA section end) and there is the risk that the binary data have such a byte sequence. Base 64 encoding (xsd:base64Binary) is a solution because each byte is mapped to a valid 64 character “alphabet” which guarantees that a byte is not misinterpreted as anything that would break an XML parser. Unfortunately, base 64 encoded binary data are on average 33% larger (number of bytes) than the raw data. The alternative of hexadecimal encoding (xsd:hexBinary) even expands the binary data by a factor of 2.

Another idea to handle opaque data is to place them outside of the SOAP message. This technique is well-known in the Java world as “SOAP with Attachments” (SwA). The SOAP message contains a reference to the binary data which are placed at the Multipurpose Internet Mail Extension (MIME) attachment.

But the binary attachment is not part of the SOAP Envelope at all. It’s leaving it up to the message processor to retrieve the actual binary data. Standards like WS-Security couldn’t work anymore, because WS-Security can’t be used to secure attachments. Furthermore the Microsoft world was using DIME (Direct Internet Message Encapsulation) encoded attachments. It’s not a good idea that Web Service providers have to support both attachment technologies, SwA and DIME.

Since 2005 the agreed solution is MTOM/XOP. It has the efficiency of SwA, but it does so without having to break the binary data outside of the SOAP Envelope. XOP allows binary data to be included as part of the XML Infoset. When data is within the XML Infoset, it means that it operates within the constraints of the SOAP processing model. In other words, you can control the processing of the XML Infoset using SOAP headers with policies.

MTOM/XOP uses a XOP:include element to tell the processor to replace the content with the referenced binary data which are “stored” outside of the XML document at the MIME attachment part. The logic of discrete storage and retrieval of binary data becomes inherent to the XML parser. The SOAP parser could treat the binary data in a uniform way and as part of the XML document, with no special retrieval logic. Fortunately sender and receiver are not aware that any MIME artifacts were used to transport the base64Binary or hexBinary data.

Let’s take a look on how to enable MTOM/XOP on SOA Suite 11g. The real world requirement is to build a cache warming-up dictionary service for the Web Presentation Tier which has no database access. Therefore the SCA Container running on the Service Tier provides a Dictionary Entity Service which should be able to transport large datasets to the Web Presentation Tier intermediated by an OSB.

A first top-down WSDL-first implementation would cause the client to send a request for Dictionary ID ranges (request example); afterwards the service retrieves the requested dictionary data from the database and sends the response back to the client as normal dictionary entry structure (response example).

Request Example:

Response Example:

It’s obviously that a large range of Dictionary IDs would cause a large SOAP Response message. But in our case we have a Spring Bean implementation using the simple JDBCTemplate approach. The result of the database request is a list of maps (line 11 and 12) which is mapped on the service response (line 32 to 38).

How about that the service return the raw list instead of the collection of dictionary entry type? Isn’t it a technical motivated private service where the known Java client on the Web Presentation Tier has to navigate through the list anyway? Yes, it is.

The service should become MTOM enabled in order to handle the binary data as part of the message. The same request structure should return a byte array with the data list returned from the database. Two meta data information should be returned additionally, how much dictionary entries are returned and the byte size of the array.

The XSD structure is simple. Element Size and RecNum are optional and of type long. And here we use the base64Binary type for the list of dictionary entries (Element return).

MTOM will be enabled with a right mouse click on the Web Service Adapter placed at the Exposed Services swim lane. The menu option “Configure WS Policies …” will open the Web Service Policies configuration wizard. You only have to enable the MTOM policy (oracle/wsmtom_policy).

The Web Service Adapter on the graphical SCA Editor will have a lock icon at the right upper corner after adding the policy. Behind the scene the wizard will add only one additional configuration line on the Web Service binding (line 4) at the composite.xml file.

When you deploy the MTOM enabled service and take a look at the WSDL URL you will realize that Oracle SOA Suite added a MTOM policy to the WSDL (line 6 to 8). Additionally the WSDL binding has a policy reference to the MTOM policy (line 100). The wsdl:required="false" declaration means that also not MTOM enabled clients should work. Tools such as soapUI or SoapSonar should be able to invoke the service which makes sense because the service offers operations without MTOM involvement. From my experiences with soapUI it’s not working because both MTOM request properties (“Enable MTOM” and “Force MTOM”) have to be enabled always. Otherwise the service responses with an internal sever error (“Non-MTOM message has been received when MTOM message was expected.”).

Oracle SOA Suite will create a byte[] equivalence for the XSD type base64Binary. So byte[] is the object representation of XSD types base64Binary and hexBinary. The only need is to convert the Spring database result list to a byte array (by using the classes ObjectOutputStream and ByteArrayOutputStream inside of a DataMTOMUtil helper class at line 34). The two meta data values are quickly and easily determined (line 37 and 38).

The same request will now result in a more interesting response containing the XOP:include.

So the two meta data information of size and record numbers are like expected. But the dictionary result is now placed with context type “application/octet-stream” (=binary attachment) at the MIME part attachment with a given content ID. soapUI is showing you a raw view on the complete SOAP message.

As you could see, the HTTP message has the content type “multipart/related” which is used for compound documents. The type is “application/xop+xml”. The dictionary list is send as binary part of the MIME message with the content type “application/octet-stream” and the same content ID which is placed at the href attribute (cid:6df6245c6aff457d8d51db92b6707170).

You will realize an interesting effect when the binary data are below a size of 1024 byte. In this case the Web Service framework will not create a separate attachment. Instead the binary data are placed directly on the XML message.

The MTOM threshold of 1024 byte is a default value. When the potential attachment bytes are less than that threshold size, the data will be inlined. The MTOM threshold determines the break-even point between sending data inline or as attachment.

I already mentioned that MTOM/XOP will not do any kind of data compression. But it’s obvious that data compression will have a great benefit on our solution. The objToByteArray method is using standard OutputStream classes. It’s a quick implementation to use the java.util.zip.GZIPOutputStream for additional compression (line 10).

The standard package java.util.zip provides classes for reading and writing standard GZIP format which is a popular compression algorithm. The client side creation is simply using the JDeveloper Web Service Proxy wizard. The wizard creates a Web Service client skeleton which is also MTOM/XOP enabled.

Regarding MTOM/XOP you only have to enable the “oracle/wsmtom_policy” during the Web Service client creation.

Finally you place your code snippets at the created Web Service client skeleton. In our case the conversation of a byte array back to a List object (line 31) and the mapping to the dictionary structure (line 33 to 38). The code has also to consider the GZIP decompression (line 7).

What is the effect of data compression? I executed the service with several dictionary ranges and put the values on a spreadsheet. Additionally I added the size for base64 encoding (which would be used without MTOM/XOP).

Dictionary Rec Num	Dictionary List Size Uncompressed (byte)	Dictionary List Size GZip Compressed (byte)	Dictionary List Size Uncompressed Base64 (byte)	Dictionary List Size Compressed Base64 (byte)
1	646	493	864	660
14	2706	740	3608	988
194	36383	3737	48512	4984
1783	307619	13830	410160	18440
11574	2188494	89687	2917992	119584

The compression rate is significant. Take a look at the last line. When I would transport 11574 records without MTOM on the base64Binary element I have to base 64 encoding of the data. In this way I need 2.78 MB byte (2917992 byte) which is 25% more than the original dictionary list size of 2.09 MB (2188494 byte). A GZIP compression will bring down the original dictionary list size at 0.09 MB (89687 byte). This is the final size which travels on the MTOM/XOP MIME attachment which is around 4.1% of the original size.

Therefore service is able to transport much more data in a given time with the GZIP compression. Base64 encoding will add additional size and is not a good option (even when you compress the dictionary list because it’s still around 25% larger). So MTOM/XOP with GZIP compression is the best solution.

The solution shown happens in memory. The Oracle documentation describes also a streaming approach which may be interesting in case of very large binary data. But the documentation also mentions limitations regarding message encryption which is not working on streaming MTOM.