BizTalk 2010 Custom Pipeline Component -Using XPathMutatorStream with MarkableForwardOnlyEventingReadStream

The XPathMutatorStream could be used together with the MarkableForwardOnlyEventingReadStream on the BizTalk Custom Pipeline Component to situations like evaluating  the content of the message based on XPath .
Below is the code snippet .
 public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage InMsg)
        {
           
            // get the original stream into an instance of MarkableForwardOnlyEventingReadStream 
            MarkableForwardOnlyEventingReadStream stream = new MarkableForwardOnlyEventingReadStream(InMsg.BodyPart.GetOriginalDataStream());
           
     //Create an XPathMutatorStream instance
            XPathMutatorStream xpathStream = GetXpathMutatorStream(stream);
            InMsg.BodyPart.Data = xpathStream ;
            pContext.ResourceTracker.AddResource(xpathStream);
            return InMsg;
            ////
        }
  private XPathMutatorStream GetXPathMutatorStream(Stream OriginalStream)
        {
            // Define the Xpath Collection
            XPathCollection XpathQueries = new XPathCollection();
  //you can declare namespaces to get rid of the "ns0" prefix issues as well
            //XpathQueries.NamespaceManager  = new System.Xml.XmlNamespaceManager();
            //XpathQueries.NamespaceManager.AddNamespace("ns0","https://blabla/");
  //Xpath Expressions
            XpathQueries.Add(new XPathExpression(FormatXpathQuery("/InMsg/InField1/")));
            XpathQueries.Add(new XPathExpression(FormatXpathQuery("/InMsg/InField2/")));
            // Add the Mutator to this Stream
            ValueMutator XPathDelegate = new ValueMutator(this.OnXPathMatch);
            XPathMutatorStream xpathStream = new XPathMutatorStream(OriginalStream, XpathQueries, XPathDelegate);
          
            return xpathStream;
        }
  private void OnXPathMatch(int matchIdx, XPathExpression xpathExpr, string ActualValue, ref string ReplaceWith)
        {
            // Any logic which needed to be applied on the value returned from the Xpath .
            if (ActualValue == "quack-quack")
            {
                ReplaceWith = "Duck";
            }
            if (ActualValue == "qluck-qluck")
            {
                ReplaceWith = "Hen";
            }
        }

Sending Flat File over AS2 - A Practical Guide

This article focuses on how to send a flat file message from BizTalk to the Trading Partners using AS2 .

I am going point-blank to focus mainly on the steps rather than the concepts.

Ingredients required from the Sender:

- BizTalk Server 2010 with EDI/AS2 Runtime configured - we need this to send and receive messages(inc EDI) using AS2

- Signing (Signature verification) certificate - we need this for the sender to sign the message and receiver to verify the signature. The public key part of the certificate should be shared/configured at the receiver end.

- AS2 ID - any unique ID specific to the sender . This should be shared to the receiver.

Ingredients required from the Receiver:

- an AS2 software  - to receive& process AS2 message/send MDN to the sender.

- Encryption certificate - public part will be shared with the sender.

- AS2 ID - any unique ID specific to the receiver.This should be shared to the sender.

- an HTTPS site , to receive the AS2 message , the sender will the HTTPS URL to send the AS2 message .

In common , the server IPs from both sender and receiver should be firewall configured to allow the network traffic.

 BizTalk supports RC2 and DES3 encryption algorithms by default.

Assuming that the Flat flat is already available from the biztalk orchestrations /pipelines , it is time to set up the Trading partners and AS2 details to send the flat file message.

Create Parties & Send Ports

- create a send port (2-way if MDN is expected) with HTTP ,configuring the HTTPS url provided by the receiver.for ex (https://abc.com/welcome/toas2). There are few things to note while configuring the HTTPS adapter , such as configuring the certicate and thumbprint so on.

Please refer to the below url on configuring HTTPS adapter.

http://www.codeproject.com/KB/security/HTTPBizTalk2009Win2008R2.aspx

- create a new Party for the sender from the BizTalk Admin console ,with a name (ex: ABCLTD) and make sure to tick the box for "Local BizTalk processes messages received by the party or supports sending messages from the party" - This tells that the sender party is using BizTalk for sending messages to other parties.

- Similarly create a new Party for the Receiver (ex: XYZLTD).

Creating the parties ,also creates a profile for each party ,which we will use to create new Agreement between the 2 parties.

- Create a Agreement by right clicking on the sender party's (ABCLTD) profile, this will bring up the Agreement properties dialog box.

- You can specify a name for the agreement such as AS2_ABC_XYZ

-  Select the Protocol as AS2, from the drop down.

- on the second party (the receiver), select the XYZLTD party created earlier.

- on the ABCLTD ->XYZLTD tab, provide the AS2 ID for the sender and receiver into AS2-From and AS2-To respectively.

- on the validation ,tick "Message should be signed" & "Message should be encrypted" with the encryption algorithm either as RC2 or DES3 as agreed between the 2 parties.

- on the Acknowledgement (MDNs) , tick "Request MDN"  & "Process inbound MDN into MessageBox for routing/delivery options", if you wish to receive MDN back from the receiver.Also, on the Disposition-Notification-To box , enter any value ,even meaningless (not sure y this is been made as mandatory !?!) .The MDN received will go directly into messagebox ,so that any other send port can be used to subscribe to that.

- on the Send Ports , select the Send port created at the begining ,which will be used to send the flat file message to the receiver,via the https url.

- on the XYZLTD -> ABCLTD tab, provide the AS2 ID for the sender and receiver into AS2-From and AS2-To respectively.

- on the Validation, tick "use agreement settings for validation and MDN instead of message header",

"message should be signed" &  "message should be encrypted" along with RC2 or DES3 for the algorithm.

So by now , we have set up the Parties , user profile and the AS2 agreement between them .

Once the Biztalk application is deployed with the orchestrations and pipelines (to send a flat file , we need to have a custom flat file assembler with AS2 encoder ), we can conclude that it is ready for testing.

This article provides at the very basic level ,with much more details on working with AS2 from BizTalk and hope that this will be a starting point for anyone ,who would struggle to set up AS2 within BizTalk.

Regards

Sullu



BizTalk AS2 EDI

This article will briefly discuss about the AS2 & EDI support in BizTalk 2010.

Unlike the earlier versions, BizTalk 2010 has extensive native support for EDI message processing & also AS2 protocol support.

BizTalk 2010 introduces new concepts for Trading Partner Management (TPM) such as Partner profiles & Agreements. We would now be able to define the trading partners involved in the B2B process as Parties in BizTalk with separate business profile created for each party.Seperate Agreements could be created for different protocols such as EDI(X12/EDIFACT)/AS2  between two different parties .

The reason behind Business profile is, to have different profiles created for different units/teams within the same Party, so that different units/teams within the same parties can have different agreements defined on different protocols.

For example Company A has 2 business profiles created for Sales & Marketing units.

Let’s say, Company A – Sales unit connects with Company B - Sales unit using AS2 /X12 and Company A – Marketing unit connects with Company B - Marketing unit using AS2/ EDIFACT, then there will be separate Agreements created for each business profile.

Now , we know the terminologies of TPM such as Parties, Business Profiles, Agreement, Message Protocols (EDIFACT, X12) & communication protocol AS2.Its time to see the steps involved in creating & configuring a basic TPM solution between 2 parties (Let’s say Company A & Company B).

1.     Decide the message protocol .I am going with X12 for this article. But, there are only few differences between X12 & EDIFACT.

2.     Decide & list the type of transactions & documents between the parties. BizTalk provides schemas for almost all of the transaction types. I will go ahead with X12 04010 856, which is the shipment notice schema.

3.     Decide the format of transactions (XML, EDI).This will make a difference in choosing the right Pipeline in the send/receive ports. The default pipelines available for AS2 processing are AS2Send/AS2EdiSend & AS2Receive/AS2EdiReceive respectively for XML and EDI.If you want to generate the message in edi format then we should use AS2EdiSend, otherwise it will be AS2Send.

4.     Define the mapping rules for the EDI transactions. This step is very important and complicated during the entire process, as we should have the mapping implemented properly in BizTalk to generate the EDI message, as desired.

5.     Create & configure TPM (Parties, Business Profiles & Agreements).Parties and business profiles will be created for CompanyA & CompanyB .Separate agreements will be created for AS2 & X12 .The AS2 agreement will have configuration settings to specify whether the messages needs to be signed/encrypted/compressed, whether to request for MDN and also to associate the send port, which will be used to send the message to CompanyB. Based on the settings for signing & encryption, certificates should be configured at send port level.

6.     Once all above are done, we can create & deploy the BizTalk artefacts needed for the entire process.

This might include

-         Creating internal shipment schema for CompanyA

-         Adding the existing X12 04010 856 schema provided by BizTalk.

-         Create mapping between the two schemas.

-         Create orchestration, if it is necessary. Otherwise, the mapping can be configured in the send port.

-         Create & configure the necessary physical receive & send ports.

-         Build & deploy the project.

So, briefly, CompanyA creates internal shipment notice which matches its internal schema & the same is mapped against X12 04010 856 schema. The generated EDI message is validated, assembled & encoded by the AS2EdiSend pipeline and delivered to the CompanyB‘s HTTP/HTTPS url configured in the send port. The AS2 specific message context is included from the Agreement defined between the 2 parties. The Agreement also helps in specifying the Transaction set validation rules, character set and separators and so on.

If we need to receive an MDN back from the CompanyB, for the X12 04010 856 message sent, we can configure the same in the Agreement and with the help of 2-way send port, the MDN can be received in BizTalk.

Hope this article helps to understand the basis of AS2/EDI processing with BizTalk 2010 & the high level steps involved in implementing a solution. Thanks for reading & Good Luck!!!

My Initial reading(All I know so far) on WCF-SQL Adapter with BizTalk 2010

In this article, I am going to briefly describe about the various benefits of using the new WCF-SQL Adapter, released as part of the BizTalk Adapter Pack 2010.

I have been working on SQL Adapter with the BizTalk 2004/2006 versions & all I used to use is generating the SQL schema from the stored procedures for the DML operations (SELECT, INSERT, UPDATE & DELETE) .I have been hit with few issues during my personal experience with SQL Adapter. Few of them,

·         The schema generated using SQL adapter is not very flexible.Means, I can’t promote or distinguish fields. Factors include, the adapter generates them as either attributes or record nodes. I need to tweak the schema manually, to convert them into elements & then I can either promote or distinguish.

I am not going to compare the SQL Adapter with WCF-SQL, just for the above mentioned issues.

But I am going to describe the features & benefits of WCF-SQL over SQL Adapter.

WCF-SQL Adapter Features

·         SQL Server DB can be exposed as WCF service to external clients – exchange of SOAP messages for request-response.

·         Support for performing DML operations directly on to the SQL tables. The schema generated will have the same number of fields as the table & u have the option to generate separate schema for separate DML operation. These schemas make it very simple and easier for mapping SQL to/from other message formats.

·         Support for strongly typed stored procedure. In the SQL Adapter, the stored procedure should return the result set as XML AUTO, XMLDATA .otherwise, we will have pain in generating the schema. With WCF-SQL adapter, we can generate schemas from stored procedures without FOR XML. Also, the schema generated will have the list of fields which are returned in the result set. For example,

Select ID, NAME, DESC, AMNT from Table where ID = @ID, inside the stored procedure.

Then, the schema generated will have the fields as ID, NAME, DESC, AMNT.This is lot more easily not to do mapping & message construction within BizTalk orchestration.

Also, we can generate schema with <Any> field instead of the specific field names. This is the difference between strongly typed stored procedures & stored procedures.

·         WCF-SQL adapter also provides backward support for the stored procedure with FOR XML AUTO.

·          The CLR type ADO.NET functions like ExecuteNonQuery, ExecuteReader & ExecuteScalar are also supported by WCF-SQL adapter. These functions become useful for situations like INSERT, UPDATE, and DELETE where you won’t expect any result set back, but an integer return value & if you want to return single value from SQL (value from 1^st row 1^st column).Another advantage of using these functions is, executing multiple SQL statements.

·         Composite operations – means we can execute related set of operations together on different database tables (SELECT is not supported). This means you can perform separate insert & delete operations together with the composite message. All we need to do is to construct a composite schema with all the operation schemas referenced.

WCF-SQL adapter is not just whatever I have listed above. It has much more feature sets & options for developers such as polling based on specific conditions, SQL-WCF adapter using WCF-Service & channel model and so on. The Binding for the WCF-SQL adapter is been made easy (binding file generated along with the schema), which can be imported to configure physical send/receive ports.

I will try to cover up and advanced version of this article, with elaborating more on the WCF-SQL adapter, with some hands on samples. Thanks for reading & happy Biztalking!!!

BizTalk Tips & Tricks

In this article you can find some of the tips & tricks ,best practices learned and applied during my career experience.I hope readers will find them useful and I am happy to get any comment/feedback & objections on any of my posting.Thanks & Happy BizTalking.

1. BizTalk Best Practices

Always use multipart messages in Orchestration

o    If schema changes (replacing with new schema), no need to change the orchestration send/receive.

o    Practical approach – if the schema is changed, delete the part from the multipart message. Lot of time saved. No needs to touch receive/send shapes & receive/send logical port.

·         Always use Direct Binding

o    Messages are always processed using subscriptions. So, how does it matter?

o    Loosely coupling.

o    Reusable orchestrations

·         Always use Canonical Schema (Separate Internal & External Schemas)

o    One single canonical schema can have everything in it .Ex: Order schema can have all the order details at one place. Anybody and everybody can refer to the same schema to pull out any data.

o    Flexible. If partner schema changes, only corresponding Map needs change. Orchestration unaffected.

o    Generic Orchestrations can be created with canonical schema.

·         Always Deploy at Solution Level in VS ,rather than project Level

o    Automatic build/deploy of dependencies.

·         Always use straight forward Maps either in Receive/Send port. Only if there are multi level input/output mappings and message constructions are needed within the orchestration.

o    No need to update Orchestration, if schema   changes.

o    Alternative - ESB (dynamic mapping -My personal choice)

·         Never expose Orchestration as Web/WCF service. Always expose schemas to the caller

o    Loosely coupling

o    Changing orchestration without breaking the caller.

·         Always use relative paths for Config & Key files

o    Avoids source control build issues!

o    Practically – relative path “../” means one folder up, generally from the bin folder or in BizTalk bin/development folder.

·         Debug XSLT used in Maps separately using VS Debugger.

o    Maps are not debugger friendly

o    Input can be validated against the mapping rules and helps to fix any mapping error prior to deployment.

2. Some useful BizTalk Tips

To Debug XSLT

·         Validate the Map

·         Open the xslt generated from the validate map (generally in C:\Users\<Porifle>\AppData\Local\Temp\_MapData\ folder)

·         Right click – View source

·         Set Break points

·         Set input XML document, which you want to use to validate the Map, in the properties

·         set the output XML path. This will be result generated once the debug is complete.

·         Menu – XML – Debug XSLT

·         Verify the output document – check you have got everything u needed.

·         If any error, you will be shown them in the Output window.

·         That’s it done.

Direct Binding Orchestration

My Ideal way of creating Orchestrations.

·         Receive shape direct binding message box

·         Send shape in direct binding message box

·         Receive shape can have filters – such as Receive Port/Location Name or any other promoted properties

·         Send port can have filters such as Message Type

    3.Useful Information on BizTalk  

Schema versioning

·         Multiple versions of the same schema (same message type = root node +namespace & same dll name, but different version such as 1.0.0.0 /2.0.0.0) can be deployed in the same server .But; the latest version will be picked up by the BizTalk.

MinOccurs vs. Nillable (Optional vs. Not Required)

·         Optional -field doesn’t need to be present in the XML.

·         Required

·         Not Optional

·         Not Required - field doesn’t need to have a value.

Possible combinations:

·         Optional + Not Required

·         Optional + Required

·         Not Optional + Not Required

·         Not Optional + Required

MinOccurs = 0 means optional.

Nillable = “true” means not required.

String data types are Nillable by default

Other types – integer, bool, date, time etc are required by default – to make it not required set Nillable = “true”

BizTalk Orchestration Profiler – tool to get the coverage on the orchestration tracking. Used to identify long running shapes, low latency issues .gives number of instances run over a period of time and the coverage at shape level.

BizTalk Documenter – Tool to create documents in chm /word format with the BizTalk artefacts details in the assembly.

XSDObjectGen.msi – Generates .NET class of schema equalent.similar to xsd.exe.

4.BizTalk – Don’ts

·         using XmlDocument in custom pipeline component . XmlDocument consumes more memory.Use streaming instead.

·         using external schema directly in internal mapping & processing . maintain equivalent internal schema will reduce lot of rework even if the external schema undergo changes.