BizTalk WCF-WebHttp adapter does not detect 500 error

Update 2016-11-17: Microsoft has fixed the bug in CU5 for BizTalk 2013 R2: https://support.microsoft.com/en-us/kb/3121493

Original post:

For a project at my current client, we use a REST-based API to retrieve some data using the BizTalk 2013R2 WCF-WebHttp adapter. We tried this in a message-only scenario, but soon noticed that errors did not give the desired result. It seems the port sending the response from the API back to the caller did not realize that an error had occurred. In other words: the WCF-WebHttp adapter does not detect all errors.

To further debug this behavior, we tried to call the same API using a simple orchestration. In this orchestration a request is received from a Filedrop location, mapped to the request for the API (to promote the required properties for parameter mapping in the port) and sent to the MessageBox using a two-way send port. Around the send and receive shape is a scope with an exception handler on System.Exception. The exception that occurs is then output via a System.Diagnostics.Trace() statement.

To further assess the behavior, the WCF.InboundHttpStatusCode context property of the API response message is also written to the trace.

Finally, the response of the API is sent to disk via a FILE send port.

The part of the orchestration that does the REST API call is depicted below:

OrchestrationWCF-WebHttpAdapterTest

If we test this solution with a happy flow message, all is well.

If we test this solution, with a call that gives a 400 error in the API, it also works: the exception handler kicks in and the error is written to a trace statement.

However, if we test this solution with a call that gives a 500 error in the API, the orchestration thinks that the call was successful and will not enter the exception handler. There is no data in the response message, only an error. In a regular orchestration, not a test like this one, we would now try to process the data received and fail. Also, in a message-only scenario this will cause unexpected behavior.

If we check the HTTP status code in the different scenario’s, we find the following:

In case of a HTTP 200 StatusCode, the ContextProperty value is “OK”.
In case of a HTTP 400 StatusCode, the ContextProperty is not available.
In case of a HTTP 500 StatusCode, the ContextProperty value is “InternalServerError”.

We would expect that the BizTalk engine would recognize a HTTP 500 error and act on it, but it does not. As a work around you could develop orchestrations to check for the WCF.InboundHttpStatusCode ContextProperty and its value.

Since I am unable to find anywhere that this is the expected behavior, I assume this is a bug in the WCF-WebHttp adapter. A bug report is on its way to Microsoft, so hopefully it will be fixed soon. Does anyone find the same behavior or is this the behavior that is expected?

Update 2016-07-12: Microsoft support has responded and the product team has verified the bug. However, since not a lot of users are affected and we have provided them with a workaround, they do not intend to fix it at the moment. If you have the same issue, please report it to Microsoft as well, so they are more motivated to fix it :). You may refer to our bug registration: (number removed, since it is fixed).

Check for extra fields in XML with BizUnit

With BizUnit it is easy to create checks on the values of XML nodes. It is also easy to add a count for specific repeating elements. However, it is difficult to check if there are unexpected nodes or attributes in the message, without creating counts for all nodes/attributes in the message.

Today I came up with an easy way to verify if no nodes or attributes are added (or removed, for that matter).

The xpath descendant::node() will flatten your XML (both elements, values and attributes). Add a count around this and you are set to verify if any unexpected nodes or attributes are in the XML. Below is what your XPathValidation node would look like – just add it at the bottom of your ValidationStep:

<!-- Check for unexpected nodes in the XML -->
<XPathValidation query=”count(descendant::node())”>123</XPathValidation>

You could get the count with DanSharp XML Viewer or by running the test once and checking the found value.

Getting a HTTP 401 on an external WSDL that is on a network share

At my current client we have a few exported BizTalk WCF services that are stored on a network share (on a different server then the IIS server). These WCF services have an externalMetadataLocation specified and Anonymous  Authentication is enabled. The external WSDL is in the same folder as the .svc file.

When browsing these services, the .svc files are loaded without a problem. However, when you click on the link to the WSDL, you will receive a 401 error ‘you do not have permission to view this directory or page using the credentials that you supplied’.

It turns out that, by default, IIS is using the local IUSR to browse the WSDL when Anonymous Authentication is enabled. For some reason, the svc file will load fine, but the WSDL will give the 403 (the local IUSR account is not known on the network share).

You can solve this via the IIS Manager.

  1. Go to the features screen of the application
  2. Open the Authentication settings
  3. (right)Click Anonymous Authentication and choose ‘edit…’
  4. Select the ‘Application pool identity’ option and save the settings
  5. Restart IIS and you should be able to browse the external WSDL.

 

Error when publishing a message from a convoy in BizTalk Server 2013

Today I was migrating another interface from BizTalk Server 2009 to BizTalk Server2013. This interface included a sequential convoy with a filter on the activating receive shape. Multiple MessageTypes can be processed and all shapes and ports use the XmlDocument type to communicate with the MessageBox. When publishing a output message from within the loop, the message has the same context property with the same value as the filter in the activating receive shape of the orchestration. Because the activating receive shape starts a correlation, the subscription also includes a few “exists” checks, beside the mentioned filter. This is why the outgoing message is not picked up by the same orchestration again – the published message does not have the other context properties in the subscription of the activating receive shape.

All this works as expected in BizTalk Server 2009, but in BizTalk Server 2013 an error occurs when publishing the message to the MessageBox. The orchestration is suspended with the following error message (no routing failure occurs):

Uncaught exception (see the ‘inner exception’ below) has suspended an instance of service ‘ConvoyPublishingErrorTest.ConvoyOrch(70c08f5c-8d53-9a89-87f3-5dd970d8daba)’.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: 26d22144-3130-4a5e-8da8-76356c646326
Shape name: Send_OutgoingConvoy
ShapeId: 2c8a68fb-8bcc-4551-b87b-836e8151fbf7
Exception thrown from: segment 2, progress 4
Inner exception: Exception occurred when persisting state to the database.

Exception type: PersistenceException
Source: Microsoft.XLANGs.BizTalk.Engine
Target Site: Void Commit()
The following is a stack trace that identifies the location where the exception occurred…..

In the Eventviewer, the following error by the source ‘BizTalk Server’ is shown:

The following stored procedure call failed: ” { call [dbo].[bts_AddConvoySetInstance]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}”. SQL Server returned error string: “The statement has been terminated.;Cannot insert the value NULL into column ‘vtProp1’, table ‘BizTalkMsgBoxDb.dbo.ConvoySetInstances’; column does not allow nulls. INSERT fails.”.

Apparently something is changed in the handling of convoys, causing our solution to fail. To demonstrate this behavior I have created a little demo solution you can download below. Deploy this solution on your BizTalk Server 2013 environment, import the bindings, create the filedrops, start the application and use the provided Input.xml message to see what goes wrong (copy/paste the input message, since the receive location monitors “* – Copy.xml”). All the expression shapes in the orchestration are trace outputs, so you can use DebugView to monitor the orchestation.

We will open another call with Microsoft for this problem. As soon as we hear something, I will update this blogpost.

Download demo solution.

Update 21-01-2014: After some research by Microsoft, a workaround is found for this issue. This workaround includes setting an extra != filter on the activating receive shape.

To fix the issue in the demo solution, I have added a third ContextProperty to the PropertySchema. I promoted a random field from the InputSchema into this property and set the filter on the activating receive shape to <newContextProperty> != “ValueItWillNeverHave”. As long as the incoming message does not have “ValueItWillNeverHave” as a value for the promoted field, the orchestration will pick up the received message. On the message that will be sent out, I add the ContextProperty in the MessageAssignment with the value “ValueItWillNeverHave”. With this explicit filter (as apposed to the implicted filter we had before), the exception no longer appears and the convoy keeps running as expected.

However, we feel this is basic BizTalk Server functionality that should work as it did in previous versions of BizTalk Server. Therefore we have asked Microsoft to further investigate this issue and solve it at the core. This has to be done by the Product Team and might take several months before we hear more about this. I will update this post if we receive more information. We have also asked Microsoft to add this issue to the known issue list of BizTalk Server 2013, so it is documented for other users.

Incorrect validation error message when using the default XMLReceive pipeline in BizTalk 2013

At my
current project we are migrating from BizTalk Server 2009 to BizTalk Server
2013. We had a few interfaces using the default XMLReceive pipeline for the validation
of incoming messages. These interfaces had BizUnit tests, that would check the
Error Message in the Error Portal. After the migration, these tests failed,
because the Error Message was different. We found that the error was not what
we expected and was incorrect.

Below an
example to show the behavior we found:

A simple
schema had been defined, with three string elements: FieldA, FieldB and FieldC.
FieldB has a maximum length of 5 characters.

This schema
has been deployed to BizTalk and a FILE receive location is created. This
location uses the default XMLReceive pipeline with validation enabled and the
fully qualified name of the schema set at the DocumentSpecNames.

The
following message will be used to test the validation. FieldB had been filled
with a value of eight characters (which is three too many).

We would expect BizTalk to give the following
error message:

There was a
failure executing the receive pipeline: “Microsoft.BizTalk.DefaultPipelines.XMLReceive,
Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35” Source: “Pipeline Receive Port:
“ValidationReceiveError” URI:
“C:FiledropValidationReceiveErrorIN* – Copy.xml” Reason: The XML
Validator failed to validate.
Reason: The document failed to validate because of the following error:”The ‘FieldB’ element has an invalid value according to its data type.” .

Instead, we
receive the following error message:

There was a
failure executing the receive pipeline:
“Microsoft.BizTalk.DefaultPipelines.XMLReceive,
Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35” Source: “Pipeline ” Receive Port:
“ValidationReceiveError” URI:
“C:FiledropValidationReceiveErrorIN* – Copy.xml
Reason: The document failed to validate because of the following
error:”The element ‘FieldB’ cannot contain child element ‘FieldC’ because
the parent element’s content model is text only.” .
 

This
behavior has been found on different BizTalk machines, with and without CU2
installed. We have also tried different types of validation errors and all of
them give a different error message than the actual error in the XML.

Has anyone
found this behavior as well? If we create a custom pipeline with a XML dissassembler
and a XML validator component, we receive the error we expect. We will open a
call with Microsoft and I will update this post if we receive or find a
solution.

BizTalk BAM event not written from PipelineComponent when port fails

When developing a custom PipelineComponent I spend quite a while overlooking the obvious. For all those who also overlook this:

When writing BAM from a PipelineComponent it is a good practice to use the PipelineContext.GetEventStream() method to get the BAM eventstream. This way your BAM events will piggyback ride along on the SQL calls already made by the BizTalk messaging engine.

Firstly I wrote a little console app to create some BAM instances from code using a DirectEventStream. When this gave the results I wanted, I developed the PipelineComponent using the same code, but with the MessagingEventstream instead of the DirectEventStream. After I wrote the PipelineComponent and created a little sample with just a receive port (which would cause a routing failure), the BAM instance did not show up in the database.

I was not sure if the PipelineContext.GetEventStream() method worked correctly, so I tried to use the DirectEventStream in the PipelineComponent to see if that would work. It did. After that I added some Debug statements in the component using the correct EventStream and ran it again. All the debug statements were hit, but nothing showed up in the database.

Then it suddenly dawned on me… Since the port was failing due to the routing failure, maybe the transaction of the MessagingEventstream was cancelled? After adding a send port to solve the routing failure, the next BAM activity did show up.

So, if you use the PipelineContext.GetEventStream() method it will only store information if the entire receive port was executed succesfully. If this is your requirement, that is fine, but if you want to store everything that goes through the pipeline, don’t use the MessagingEventStream.

Update: after the comment by Thiago Almeida I tried the same for a send port. In a send port the BAM event is written even if the send port fails. Also a thing to keep in mind.

BizTalk: Two versions of a schema side by side and using them in orchestrations

Schema versioning in BizTalk is always a difficult thing to do. When small changes are done on a schema (for a new solution), but this schema is already in use by a solution which is currently in production, you have to be very careful.

BizTalk defines the schema type of a message by the combination: targetnamespace#rootnode to make a MessageType unique (for example: http://mynamespace.com/#MyRootNode). This will define the type of the message and is also used in subscriptions.

If you deploy two schemas with the same MessageType in different assemblies and you try to receive an instance with this MessageType, BizTalk will give you a routing failure telling you it is not able to determine the exact MessageType (the MessageType is found multiple times in the database).

Suppose you have schema X in DLL Y version 1.0.0.0 and you fix a bug (change a nodes’ default value for example) for a new version of a solution, and increase DLL Ys version to 1.0.1.0. If you deploy this new version of DLL Y side by side with an old version, BizTalk is smart enough to see that it is a newer version and will use this newer version of schema X for received messages with this MessageType. No routing failure! This works excellent in a message-only scenario.

So far, so good. If you use the MessageType in a receive shape in an orchestration, this orchestration has a subscription on this MessageType. That is what you expect and see in the ‘View subscriptions’ panel of the Administration Console:

As you can see, there is no assembly version shown here! But since the orchestration is developed with a certain version of a schema, the actual orchestration code does expect a certain version of the schema for the received instance. Meaning that the subscription will work, but the orchestration can’t use the message, because it was typed as the 1.0.1.0 version and the orchestrations expects the 1.0.0.0 version. This will generate an error in the Administration Console as follows:

Uncaught exception (see the ‘inner exception’ below) has suspended an instance of service ‘VersioningTestBTProjOrchv1000.BizTalk_Orchestrationv0100(e0f5b7e2-7e31-550e-1022-af218b9002bc)’.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: 8b5f3dd1-42b6-4bb0-b7a4-ecd87ced4813
Shape name: Receive_1
ShapeId: 4747131a-9c71-46e6-a99a-319053adcfe7
Exception thrown from: segment 1, progress 3
Inner exception: Received unexpected message type ‘VersioningTestProject.Schemas.SchemaIncoming_v0100, VersioningTestProject, Version=1.0.1.0, Culture=neutral, PublicKeyToken=b5c9ca1ed57903d0’ does not match expected type ‘VersioningTestProject.Schemas.SchemaIncoming_v0100, VersioningTestProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b5c9ca1ed57903d0’.
       
Exception type: UnexpectedMessageTypeException
Source: Microsoft.XLANGs.Engine

BizTalk does not automatically know for which orchestration to use the old version and for which the new version. You could say that BizTalk should be smart enough to see that an orchestration obviously expects the older version and cast the message to the correct version before feeding it to the orchestration. But this might lead to unforeseen behaviour and probably is not done this way to avoid errors.

There is an easy fix for this problem. You probably have a different receive location for the newer version of the schema, since the old version is running side by side. You can tell BizTalk to cast a received message to a certain version of the schema. This way it will not automatically cast the message to the latest version, but it will cast it to your specified version. This can be done on the XMLReceive pipeline as follows:

1 – Open the XMLReceive pipeline properties:

2 – Enter the fully qualified name of the type you are going to receive as the DocumentSpecNames property.

The name should look like: ‘FullSchemaTypeName, FullAssemblyName’ (for example: VersioningTestProject.Schemas.SchemaIncoming_v0100, VersioningTestProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b5c9ca1ed57903d0). This information can be found by double clicking the deployed schema in the BizTalk Administration Console.

Voila, now BizTalk will cast your message to the specified version and your old orchestration will run just fine

Of course this only works if you have separate receive locations for all versions and you can use the XMLReceive pipeline (or any other pipeline containing the XML dissambler component), but it might just help you out. (Only tested in BizTalk Server 2006 R2)

Exposing BizTalk Schemas as a SOAP-RPC/Encoded webservice

Yesterday I had to connect a Delphi back-end system to the BizTalk 2006 R2 environment of my customer. Since this Delphi system could only call webservices exposed as SOAP-RPC/Encoded, I had to expose the needed schemas as such.

While searching on this subject I found a good post of Tomas Restrepo on consuming SOAP-RPC/Encoded webservices: http://winterdom.com/2005/07/consumingrpcencodedwebservicesfrombiztalk.

This explains how you have to work with multipart messagetypes to call a SOAP-RPC/Encoded webservice. At first I figured I would have to do something alike to expose my schemas as a SOAP-RPC/Encoded webservice. However, this is actually very easy.

With the .Net framework you can easily expose a webservice as SOAP-RPC/Encoded by setting the correct attribute on the webmethod. To achieve this you should do the following to the generated webservice code after exposing your schemas as a webservice using the default BizTalk Web Services Publishing Wizard:

  1. Comment out (or delete) the SoapDocumentMethodAttribute on the WebMethod and the SoapDocumentServiceAttribute on the class that are set by default by the wizard.
  2. Set the SoapRpcMethod attribute on the WebMethod and the SoapRpcService attribute on the class.
  3. You can set parameters on both new attributes to change the behaviour and settings of the SOAP-RPC/Encoded service

That’s it. Since the .Net Framework creates an object based on your schema of the webrequest and uses this to call BizTalk, you don’t have to change anything on the BizTalk side to expose your schema’s as a SOAP-RPC/Encoded webservice. Now that is easy!

While on the subject of exposing schemas as a webservice: the generated webservice has ‘ParameterStyle=System.Web.Services.Protocols.SoapParameterStyle.Default‘ set on the SoapDocumentMethodAttribute by default. This causes an extra node in the envelope body named like the WebMethod’s name. If you don’t want this extra node, you can set the ‘ParameterStyle’ to ‘System.Web.Services.Protocols.SoapParameterStyle.Bare‘ and the extra node is not used anymore.

Start to blog on BloggingAbout.Net

Today I have started a blog on BloggingAbout.Net. Here I will be posting my findings on Microsoft BizTalk Server and .Net development.

For the past five years I have been working with .Net (mostly ASP.NET) and Microsoft BizTalk Server and many times I turned to blogs for finding answers to tech related questions. I decided I would like to give back to the community I have turned to many times. From of today I will be posting my findings and hopefully help someone out the way blogs have always helped, and will always help, me out.

Expect the first technical post later today. Please do not hesitate to comment, so I can make this blog better!