You might have heard about MTOM/XOP. Possibly you have a broad idea of what MTOM/XOP is doing. Than you start from the same initial situation like me some months ago. Let me give you a jump start on what MTOM/XOP means in general and in particular on the Oracle platform.
MTOM is the W3C standardized Message Transmission Optimization Mechanism which is used in combination with the concrete referencing mechanism XOP which stands for XML-binary Optimized Packaging. MTOM/XOP is a way to handle opaque data with XML like binary data of images, audio or video. MTOM/XOP does not use any compression and decompression technique. There is another standard called Fast Infoset that addresses the compression issue. But large application data which need to be compressed for efficacy could be also handled by MTOM/XOP (more about it on the given example below).
XML parsers will fail when you place binary data in a text node of a XML document. The binary data could contain reserved XML characters like ‘<’, ‘>’ or ‘&’ which will break the parser on the “other side”. CDATA wrapping allows these reserved XML characters but the parser is looking for a ‘]]>’ (which marks a CDATA section end) and there is the risk that the binary data have such a byte sequence. Base 64 encoding (xsd:base64Binary) is a solution because each byte is mapped to a valid 64 character “alphabet” which guarantees that a byte is not misinterpreted as anything that would break an XML parser. Unfortunately, base 64 encoded binary data are on average 33% larger (number of bytes) than the raw data. The alternative of hexadecimal encoding (xsd:hexBinary) even expands the binary data by a factor of 2.
Another idea to handle opaque data is to place them outside of the SOAP message. This technique is well-known in the Java world as “SOAP with Attachments” (SwA). The SOAP message contains a reference to the binary data which are placed at the Multipurpose Internet Mail Extension (MIME) attachment.
But the binary attachment is not part of the SOAP Envelope at all. It’s leaving it up to the message processor to retrieve the actual binary data. Standards like WS-Security couldn’t work anymore, because WS-Security can’t be used to secure attachments. Furthermore the Microsoft world was using DIME (Direct Internet Message Encapsulation) encoded attachments. It’s not a good idea that Web Service providers have to support both attachment technologies, SwA and DIME.
Since 2005 the agreed solution is MTOM/XOP. It has the efficiency of SwA, but it does so without having to break the binary data outside of the SOAP Envelope. XOP allows binary data to be included as part of the XML Infoset. When data is within the XML Infoset, it means that it operates within the constraints of the SOAP processing model. In other words, you can control the processing of the XML Infoset using SOAP headers with policies.
MTOM/XOP uses a XOP:include element to tell the processor to replace the content with the referenced binary data which are “stored” outside of the XML document at the MIME attachment part. The logic of discrete storage and retrieval of binary data becomes inherent to the XML parser. The SOAP parser could treat the binary data in a uniform way and as part of the XML document, with no special retrieval logic. Fortunately sender and receiver are not aware that any MIME artifacts were used to transport the base64Binary or hexBinary data.
Let’s take a look on how to enable MTOM/XOP on SOA Suite 11g. The real world requirement is to build a cache warming-up dictionary service for the Web Presentation Tier which has no database access. Therefore the SCA Container running on the Service Tier provides a Dictionary Entity Service which should be able to transport large datasets to the Web Presentation Tier intermediated by an OSB.
A first top-down WSDL-first implementation would cause the client to send a request for Dictionary ID ranges (request example); afterwards the service retrieves the requested dictionary data from the database and sends the response back to the client as normal dictionary entry structure (response example).
Request Example:
Response Example:
It’s obviously that a large range of Dictionary IDs would cause a large SOAP Response message. But in our case we have a Spring Bean implementation using the simple JDBCTemplate approach. The result of the database request is a list of maps (line 11 and 12) which is mapped on the service response (line 32 to 38).
How about that the service return the raw list instead of the collection of dictionary entry type? Isn’t it a technical motivated private service where the known Java client on the Web Presentation Tier has to navigate through the list anyway? Yes, it is.
The service should become MTOM enabled in order to handle the binary data as part of the message. The same request structure should return a byte array with the data list returned from the database. Two meta data information should be returned additionally, how much dictionary entries are returned and the byte size of the array.
The XSD structure is simple. Element Size and RecNum are optional and of type long. And here we use the base64Binary type for the list of dictionary entries (Element return).
MTOM will be enabled with a right mouse click on the Web Service Adapter placed at the Exposed Services swim lane. The menu option “Configure WS Policies …” will open the Web Service Policies configuration wizard. You only have to enable the MTOM policy (oracle/wsmtom_policy).
The Web Service Adapter on the graphical SCA Editor will have a lock icon at the right upper corner after adding the policy. Behind the scene the wizard will add only one additional configuration line on the Web Service binding (line 4) at the composite.xml file.
When you deploy the MTOM enabled service and take a look at the WSDL URL you will realize that Oracle SOA Suite added a MTOM policy to the WSDL (line 6 to 8). Additionally the WSDL binding has a policy reference to the MTOM policy (line 100). The wsdl:required="false" declaration means that also not MTOM enabled clients should work. Tools such as soapUI or SoapSonar should be able to invoke the service which makes sense because the service offers operations without MTOM involvement. From my experiences with soapUI it’s not working because both MTOM request properties (“Enable MTOM” and “Force MTOM”) have to be enabled always. Otherwise the service responses with an internal sever error (“Non-MTOM message has been received when MTOM message was expected.”).
Oracle SOA Suite will create a byte[] equivalence for the XSD type base64Binary. So byte[] is the object representation of XSD types base64Binary and hexBinary. The only need is to convert the Spring database result list to a byte array (by using the classes ObjectOutputStream and ByteArrayOutputStream inside of a DataMTOMUtil helper class at line 34). The two meta data values are quickly and easily determined (line 37 and 38).
The same request will now result in a more interesting response containing the XOP:include.
So the two meta data information of size and record numbers are like expected. But the dictionary result is now placed with context type “application/octet-stream” (=binary attachment) at the MIME part attachment with a given content ID. soapUI is showing you a raw view on the complete SOAP message.
As you could see, the HTTP message has the content type “multipart/related” which is used for compound documents. The type is “application/xop+xml”. The dictionary list is send as binary part of the MIME message with the content type “application/octet-stream” and the same content ID which is placed at the href attribute (cid:6df6245c6aff457d8d51db92b6707170).
You will realize an interesting effect when the binary data are below a size of 1024 byte. In this case the Web Service framework will not create a separate attachment. Instead the binary data are placed directly on the XML message.
The MTOM threshold of 1024 byte is a default value. When the potential attachment bytes are less than that threshold size, the data will be inlined. The MTOM threshold determines the break-even point between sending data inline or as attachment.
I already mentioned that MTOM/XOP will not do any kind of data compression. But it’s obvious that data compression will have a great benefit on our solution. The objToByteArray method is using standard OutputStream classes. It’s a quick implementation to use the java.util.zip.GZIPOutputStream for additional compression (line 10).
The standard package java.util.zip provides classes for reading and writing standard GZIP format which is a popular compression algorithm. The client side creation is simply using the JDeveloper Web Service Proxy wizard. The wizard creates a Web Service client skeleton which is also MTOM/XOP enabled.
Regarding MTOM/XOP you only have to enable the “oracle/wsmtom_policy” during the Web Service client creation.
Finally you place your code snippets at the created Web Service client skeleton. In our case the conversation of a byte array back to a List object (line 31) and the mapping to the dictionary structure (line 33 to 38). The code has also to consider the GZIP decompression (line 7).
What is the effect of data compression? I executed the service with several dictionary ranges and put the values on a spreadsheet. Additionally I added the size for base64 encoding (which would be used without MTOM/XOP).
| Dictionary Rec Num | Dictionary List Size Uncompressed (byte) | Dictionary List Size GZip Compressed (byte) | Dictionary List Size Uncompressed Base64 (byte) | Dictionary List Size Compressed Base64 (byte) |
| 1 | 646 | 493 | 864 | 660 |
| 14 | 2706 | 740 | 3608 | 988 |
| 194 | 36383 | 3737 | 48512 | 4984 |
| 1783 | 307619 | 13830 | 410160 | 18440 |
| 11574 | 2188494 | 89687 | 2917992 | 119584 |
The compression rate is significant. Take a look at the last line. When I would transport 11574 records without MTOM on the base64Binary element I have to base 64 encoding of the data. In this way I need 2.78 MB byte (2917992 byte) which is 25% more than the original dictionary list size of 2.09 MB (2188494 byte). A GZIP compression will bring down the original dictionary list size at 0.09 MB (89687 byte). This is the final size which travels on the MTOM/XOP MIME attachment which is around 4.1% of the original size.
Therefore service is able to transport much more data in a given time with the GZIP compression. Base64 encoding will add additional size and is not a good option (even when you compress the dictionary list because it’s still around 25% larger). So MTOM/XOP with GZIP compression is the best solution.
The solution shown happens in memory. The Oracle documentation describes also a streaming approach which may be interesting in case of very large binary data. But the documentation also mentions limitations regarding message encryption which is not working on streaming MTOM.
No comments:
New comments are not allowed.