Piotr Woloszyn

Joshua Watkins, Piotr Woloszyn

Just before Christmas Josh Watkins and me had to face the interesting task of removing the dependency on Microsoft Web Services Enhancements (WSE) 3.0 in one of the projects that we are working on currently. The task consisted of two parts: signing the soap requests and validating incoming soap responses. Thanks to Robbie’s spike for signing requests we managed to quickly finish the first task. I might blog about this as well soon. The second one is a different story. As this turned out to be a pain in the butt and we couldn’t find easily comprehensive information about verifying XML’s digital signatures on the web we decided to share our finding with wider audience and hopefully help someone who has to face the same challenge. Alors…

Josh: I would just like to add that it was a really really … really big pain in the butt.

XML digital signatures (XMLDSIG) allow you to verify that data was not altered after it was signed. Working in the Microsoft .NET 2.0 world it wouldn’t be difficult as we could use the SignedXml class as described here:

// Verify the signature of an XML file against an asymmetric

// algorithm and return the result.

public static Boolean VerifyXml(XmlDocument Doc, RSA Key)

{
    // Create a new SignedXml object and pass it
    // the XML document class.
    SignedXml signedXml = new SignedXml(Doc);    // Find the "Signature" node and create a new

    // XmlNodeList object.
    XmlNodeList nodeList = Doc.GetElementsByTagName("Signature");

    // Load the first <signature> node.
    signedXml.LoadXml((XmlElement)nodeList[0]);

    // Check the signature and return the result.
    return signedXml.CheckSignature(Key);
}

Unfortunately the signed soap responses were hashed with SHA256 algorithm defined as:

<signaturemethod algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256">

And that algorithm is unsupported:

With the 2.0 release of the CLR, there is no way to use the RSA-SHA256 signature type with SignedXml. Adding a SHA-256 CryptoServiceProivder implemenation is high on our list of items to look at in the next version, which should enable this scenario.

- Shawn Farkas, Microsoft, 09/09/2007

Josh: When we first tried using the SignedXML method we would get a malformed Reference Element error (or something like that). Which as you can imagine, isn’t really all that helpful or descriptive.

The only available option was to verify the signature manually. Before we will get to details how to do it let’s look at the XML of the SOAP response coming back to us. As we can see there is Signature element which has the children SignedInfo and SignatureValue. SignedInfo holds information about which parts of the message have been signed, with which algorithm, and the hash of the signed element. No one will be able to tamper with the content of the nodes (in this example we sign soap:Body, wsu:Timestamp, wsse:BinarySecurityToken) without being detected through varying hash values. The SignatureValue is the content of the SignedInfo node hashed and signed with the private key of sender, so we can trust the content of that node as well. The whole problem is in the step which has to be performed before hashing: Canonicalization of the XML. Getting the process right for c14n was the most time consuming part.

Now more details and some code. Let’s assume that we implement System.Web.Services.Protocols.SoapExtension and will verify the message in overriden ProcessMessage(SoapMessage message) method, stage SoapMessageStage.BeforeDeserialize. Let’s assume as well that we have the content of the soap message as string (messageContent). To verify the digital signature of an XML document:

1. Create XmlDocument using the messageContent:

XmlDocument xDoc = new XmlDocument();

xDoc.PreserveWhitespace = false;

xDoc.LoadXml(messageContent);

2. Load the server certificate from file system:

X509Certificate2 serverCertificate = new X509Certificate2(pathToServerCert);

3. Now let’s validate message signature. Move to SignatureValue node using the lovely XPathNavigator:

XPathNavigator nav = xDoc.CreateNavigator();

nav.MoveToFollowing("SignatureValue", "http://www.w3.org/2000/09/xmldsig#");

4. The signature value requires that all of the \n inserted into the value are removed before we processed it:

string signatureValue = Regex.Replace(nav.InnerXml.Trim(), @"\s", "");

5. The signature value is saved in the messages encoded in base64. Therefore to get the actual signature value we have to convert back from base64:

byte[] sigVal = Convert.FromBase64String(signatureValue);

6. The SignedInfo XML block is extracted from the greater document. Then we take out all of the namespaces used in all of the parent nodes of SignedInfo and then reinsert them into the SignedInfo block. Without this your validation will FAIL.

Josh: And our validation did indeed fail every time without this. J

XmlNode signedInfo = xDoc.GetElementsByTagName("ds:SignedInfo")[0];
Hashtable ns = RetrieveNameSpaces((XmlElement)signedInfo);
InsertNamespacesIntoElement(ns, (XmlElement)signedInfo);

where:

public static Hashtable RetrieveNameSpaces(XmlElement xEle)
{
    Hashtable foundNamespaces = new Hashtable();
    XmlNode currentNode = xEle;
    string name = null;

    while (currentNode !=null)
    {
        //add namespace for current nodes namespace if it has one.
        if (currentNode.NodeType == XmlNodeType.Element && !String.IsNullOrEmpty(currentNode.Prefix))
        {
            if (!foundNamespaces.ContainsKey("xmlns:" + currentNode.Prefix))
            {
                foundNamespaces.Add("xmlns:" + currentNode.Prefix, currentNode.NamespaceURI);
            }
        }

        //now we add namespaces for any attributes that this node may have.
        if (currentNode.Attributes !=null && currentNode.Attributes.Count>0)
        {
            for (int i=0; i< currentNode.Attributes.Count;i++)
            {
                if (currentNode.Attributes[i].Prefix.Equals("xmlns") || currentNode.Attributes[i].Name.Equals("xmlns"))
                {
                    if (!foundNamespaces.ContainsKey(currentNode.Attributes[i].Name))
                    {
                        foundNamespaces.Add(currentNode.Attributes[i].Name, currentNode.Attributes[i].Value);
                    }
                }
            }
        }
        currentNode = currentNode.ParentNode;
    }
    return foundNamespaces;
}

public static void InsertNamespacesIntoElement(Hashtable namespacesHash, XmlElement node)
{
    XPathNavigator nav = node.CreateNavigator();
    if (String.IsNullOrEmpty(nav.Prefix) && String.IsNullOrEmpty(nav.GetAttribute("xmlns","")))
    {
        nav.CreateAttribute("", "xmlns","",nav.NamespaceURI);
    }
    foreach (DictionaryEntry namespacePair in namespacesHash)
    {
        string[] attrName = ((string)namespacePair.Key).Split(':');
        if (attrName.Length > 1 && !node.HasAttribute(attrName[0]+":"+attrName[1]))
        {
             nav.CreateAttribute(attrName[0], attrName[1], "", (string)namespacePair.Value);
        }
    }
}

7. Canonicalize the signedInfo:

Stream signedInfoStream = canonicalizeNode(signedInfo);

where:

public static Stream canonicalizeNode(XmlNode node)
{
    XmlNodeReader reader = new XmlNodeReader(node);
    Stream stream = new MemoryStream();

    XmlWriter writer = new XmlTextWriter(stream, Encoding.UTF8);

    writer.WriteNode(reader, false);
    writer.Flush();

    stream.Position = 0;

    //To ensure that the XML is properly formatted we use this transform on it
    // before creating the hash of the SignedInfo block.
    XmlDsigC14NTransform transform = new XmlDsigC14NTransform();
    transform.LoadInput(stream);
    return (Stream)transform.GetOutput();
}

8. The next step is to compute the hash of the canonicalized signedInfo:

SHA1 sha1 = SHA1.Create();
byte[] hashedSignedInfo = sha1.ComputeHash(signedInfoStream);

9. The last step of the validation of the message signature is to verify the hash using a crypto service provider via the sender’s public key:

string oid = CryptoConfig.MapNameToOID("SHA1");
RSACryptoServiceProvider crypto = (RSACryptoServiceProvider)cert.PublicKey.Key;
bool isMessageSignatureValid = crypto.VerifyHash(hashedSignedInfo, oid, sigVal);

10. OK, if we got so far it means that nobody tempered with the SignedInfo node. Now we have to validate if no one tempered with content of signed elements by inspecting references and hashes in the SignedInfo. That’s the place where the SHA256 algorithm is used. First let’s load the message references from the SingedInfo node:

XmlNamespaceManager man = new XmlNamespaceManager(xDoc.NameTable);
man.AddNamespace("soap", "http://schemas.xmlsoap.org/soap/envelope/");
man.AddNamespace("wsse", "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd");
man.AddNamespace("ds", "http://www.w3.org/2000/09/xmldsig#");

// A bit of xpath to only get the reference nodes in the security header.
XmlNodeList messageReferences = xDoc.SelectNodes("//soap:Header/wsse:Security/ds:Signature/ds:SignedInfo/ds:Reference", man);

11. For each of the references we shall perform validation (steps 12-17). First we will look up the reference listed in SignedInfo and retrieve the correspodning node from the xDoc.

XPathNavigator elementNav = xEle.CreateNavigator();
string elementID = elementNav.GetAttribute("URI", "");
//We need to remove the hash. :)
if (elementID.StartsWith("#"))
{
    elementID = elementID.Substring(1);
}

12. Now we find the node associated with the id.

XmlElement referencedNode = retrieveElementByAttribute(xDoc, "wsu" + ":Id", elementID);

where:

public static XmlElement retrieveElementByAttribute(XmlNode xDoc, string attributeName, string attributeValue)
{
    ParameterValidation.Validate("xDoc", xDoc);
    ParameterValidation.Validate("attributeName", attributeName);
    ParameterValidation.Validate("attributeValue", attributeValue);

    XmlElement foundElement = null;
    foreach (XmlNode node in xDoc)
    {
        if (node.HasChildNodes)
        {
            foundElement = retrieveElementByAttribute(node, attributeName, attributeValue);
        }
        if (foundElement == null && node.Attributes != null && node.Attributes[attributeName] != null && node.Attributes[attributeName].Value.Equals(attributeValue))
        {
            foundElement = (XmlElement)node;
            break;
        }
        if (foundElement != null)
        {
            break;
        }
    }
    return foundElement;
}

13. Then we will incorporate the namespaces from the parent node of the retrieved node

InsertNamespacesIntoElement(RetrieveNameSpaces((XmlElement)referencedNode.ParentNode), referencedNode);

14. Now we will canonicalize the node with the specified canonicalization method.

Stream canonicalizedNodeStream = canonicalizeNode(referencedNode);

15. Create the proper hash algorithm object and compute the hash of the signed node:

elementNav.MoveToFollowing("DigestMethod", "http://www.w3.org/2000/09/xmldsig#");
HashAlgorithm hashAlg = (HashAlgorithm)CryptoConfig.CreateFromName(elementNav.GetAttribute("Algorithm", ""));
byte[] hashedNode = hashAlg.ComputeHash(canonicalizedNodeStream);

16. Load the digest value and decode its value using base64 encoding:

elementNav.MoveToFollowing("DigestValue", "http://www.w3.org/2000/09/xmldsig#");
byte [] digestValue = Convert.FromBase64String(elementNav.InnerXml)

17. If the hashedNode array and the digestValue array are equal then the verification of the reference from the SignedInfo is positive! After repeating this for all the references the verification of the digital signatures of XML documents, without using WSE3 is done.

Josh: It all seems so easy now. I think the hardest part of this entire process is never really knowing if you are on the right path until you get it completely right. Hopefully this tutorial will help someone else avoid the pain and suffering that we had to endure.

Some other useful links:
Apache XML Security
An Introduction to XML Digital Signatures
Bouncy Castle

PS. REST rulez!

Popularity: 49% [?]

Share This

There are situations when you can’t put all data which you need in a single instance of a relational database. The reasons may differ. Maybe because it is too much of the data itself. Or there is a problem with network latency of a distributed architecture. Scaling? Or one of many other reasons. The answer is: horizontal partitioning. The process of splitting up your data sets that once has been called partitioning, now, it has a new name: sharding. If your data doesn’t fit on one machine you split it up into segments and each segment is called a shard. It is term used initially at Google but now spreading everywhere.

So… There is a new project Hibernate Shards which is a framework that is designed to encapsulate and minimize complexity of accessing multiple databases by adding support for horizontal partitioning on top of Hibernate Core. It was once the 20 percent project at Google but it is open-sourced now and licensed under the LGPL. If you know the Core Hibernate API you know the Shards API as its implementation hasn’t violated the Core API. Basic assumptions and paradigms for using Hibernate are still valid as Configuration, SessionFactory, and Session objects are almost exactly the same. Interfaces from Hibernate Core:

• org.hibernate.Session
• org.hibernate.SessionFactory
• org.hibernate.Criteria
• org.hibernate.Query

have shard-aware extensions:

• org.hibernate.shards.session.ShardedSession
• org.hibernate.shards.ShardedSessionFactory
• org.hibernate.shards.criteria.ShardedCriteria
• org.hibernate.shards.query.ShardedQuery

The implementations for these four interfaces serve as a sharding engine that knows how to apply an application-specific sharding logic. This logic is a set of rules how data is distributed across its shards. To specify this logic we have to implement the interfaces below:

• org.hibernate.shards.strategy.selection.ShardSelectionStrategy
• org.hibernate.shards.strategy.resolution.ShardResolutionStrategy
• org.hibernate.shards.strategy.access.ShardAccessStrategy

Hibernate Shards comes with a couple of simple implementations of these interfaces. For instance for the Shard Selection Strategy we have choice of sequential (load balanced round robin) or parallel access.

To finish the topic of the sharding logic I shall mention the id generation as well. As the standard database sequences can’t be used in distributed environment we have a choice of two primary key generators:

• ShardedUUIDGenerator - that generates a big random number
• ShardedTableHiLoGenerator - that uses a table in one of the shards to generate the primary key. Obviously it is a single point of failure for our system.

Although project is still in its early stage and even the creators warn about possible glitches this nice wrapper for hiding all the complexity of distributing data around multiple relational databases is very much worth looking at, especially if the Hibernate is already used as the ORM tool of choice.

Popularity: 35% [?]

Share This

I believe in testing, I really do. Combined with continuous integration, daily builds and automation of all aspects of build process, those are for me mandatory elements of a software project which has got any aspiration to be a successful one. Joel thinks there are more.

I believe in unit tests as well as integration and acceptance tests. I can see value of that and I use TDD happily for last two years and unit testing without being so strict about that for much longer time. It just works for me, gives me something.

However I still have questions regarding this subject. Let’s just put up first of them:

Should we or shouldn’t we unit test private methods of a class?

There are valid reasons for both of them. From one hand if we limit ourselves to a public interface and we test our class as a black box which behaves in a certain way and we don’t care how it is implemented then the tests are easier (read cheaper) to maintain. Whenever we have to change implementation we don’t have to change tests. Neat.

But there is other side as well. What about this word unit? What about TDD? If TDD can actively help us with process of developing a single unit of code why not to use that? As we all know the first red - last green routine of writing test, making it fail, developing code, making the test pass might be painful, might take long time to switch to that way of development but and the end of the day gives code of the better quality. But now what shall we do if that single unit of code by design is a private method? Shall we change that following philosophy of design for testability? It again might lead to innocent changes like converting (Java):

private void foo() {
...
}


to:

protected void foo() {
...
}

Then we can create unit tests for that method in the same package. It was really innocent. What if we have to reach to a toolbox with dirty tools? We can’t really use the same solution for C#. Let’s first have a look at the method access modifiers for that language:

• public indicates the method is freely accessible inside and outside of the class in which it is defined
• internal means the method is only accessible to types defined in the same assembly
• protected means the method is accessible in the type in which it is defined, and in derived types of that type. This is used to give derived classes access to the methods in their base class
• protected internal means the method is accessible to types defined in the same assembly or to types in a derived assembly
• private methods are only accessible in the class in which they are defined.

As we can see our dear friends from Microsoft have done it in even more problematic way. We have a choice of converting private methods to internal but our tests have to be placed in the same assembly as the production code or we can use the public modifier or we have to use some tricks with reflection. Not perfect, wrong or smells.

I understand that there is no universal answer for the question above. Applying common sense usually pays off but I wonder what is your opinion about testing in general with focus on testing private methods.

Popularity: 20% [?]

Share This

I can recall a conversation with psd from early this year about why I think current way of uploading photos to flickr sux big time. I treat flickr as repository and backup of _all_ my photos (12525 photos at the moment, still growing) and not just as a photo blog. It means that very often I have to upload quite a lot of them in batches. The existing solutions just make me angry (how shabby is the standard Flickr Uploadr!) when I have to tag them and correctly name or create sets. I am not (yet) a mac user. Maybe there is something more user friendly, I hope.

So… Coming back to the conversation with psd I have told him that unless there is some clever integration with OS which will make the process of uploading more smooth it will be always the pain. I expected as well a bit more metadata to be populated by camera automatically. How cool would be to have GPS and geo-tag all the photos when taking them. That’s not the end. I would expect as well to be able to define some tags in a camera.

That would solve a bit problem with tags. The other problems are setting permissions, creating sets etc. For pretty long time I couldn’t find any tool which I liked. Lately after migrating all my home computers to Gutsy I found something called Flickrfs as an available package. It is a virtual filesystem which mounts a flickr account as any other data file storage. It synchronizes the flickr account with local filesystem and shows photos as images files with all metadata represented as text files.

Imagine that. You can upload photos to flickr just by using cp command, the same applies to downloading photos (even if Flickr itself makes it as hard as possible to get back of your own photos). Deleting is as easy as invoking rm. You can set permissions using standard chmod command and define sets by creating symlinks with ln.

I like the idea itself very much. The Flickrfs is created and maintained mainly by one person so there might be some small “bugletes” but I still support this project and wish it the best. Well done Manish Rai Jain!

Popularity: 22% [?]

Share This

Some of you who use Eclipse 3.3 with Sun JVM might notice that it crashes from time to time. Those who have many plugins certainly experienced this more often. Problem and solution for this bug is very trivial but worth mentioning: leaking permgen space. Comes back like November flu! Just to remind the permgen space is a memory for storing “data needed by the virtual machine to describe objects that do not have an equivalence at the Java language level. For example objects describing classes and methods are stored in the permanent generation“. More information about permgen can be found in an article about tuning garbage collection.

To fix the problem edit your eclipse.ini and add following lines:

-XX:PermSize=128M
-XX:MaxPermSize=256M

You might also try Eclipse 3.3.1.1 which includes a fix for the above problem.

Enjoy!

Popularity: 32% [?]

Share This