DuraTrans

Resource Page Description
This project demonstrates how to build a service-oriented application that is a fully transactional, scalable, durable (in the event of catastrophic failure) application architecture that leverages the power of WCF and MSMQ.

The durable aspect to this project is where it really begins to shine. Applications are usually built with the assumption that they are going to work. Unit tests mean the code does what it’s expected to do, and end-to-end testing proves the application works at run-time. But what happens if the server your application is running on crashes when it’s trying to process 50 concurrent transactions? Is the data lost? Is the system in an inconsistent state? Is there a contingency plan to automatically recover?

With a durable architecture transactions are fundamental to answering all of those questions satisfactorily. Transactions ensure the system remains in a consistent state. Transactions also work symbiotically with the durable MSMQ to guarantee that messages are not lost during transmission or incomplete processing. The MSMQ can be configured to allow a pre-determined number of attempts to deliver the message. Upon failure, a contingency plan takes over and the message is sent to a DLQ (or poison queue). The DLQ is simply a service that logs the failed message so it can be handled appropriately. All this takes place within transactions, and all this takes place automatically, with no complex plumbing in your code.

The resilience of the system is in it's architecture, and the boundaries in which the services are contained. If the services are down (off-line for maintenance, re-starting, power failure, etc), this doesn't mean the system no longer works, it's merely means that those particular aspects of it are unavailable for now. It's a completely distributed system where each service/client application can run independently of everything else. The client applications can still submit message for processing, and when the services come back on-line, processing continues as normal.

The project follows a closed architecture pattern which helps to reduce internal coupling and is achieved by making each internal component (Manager, BLL, DAL) a WCF service. The benefits of WCF can now be pushed down throughout the tiers.

For example:

* system.diagnostics can now be used to trace calls throughout an application and log errors, with no change to your code.
* WCF automatically propagates the client security context across service boundaries, with no change to your code.
* Services can participate in ambient transactions (where desired), with no change to your code.
* Services can be extended by adding attributes to the method/interface, with no change to your code.

What this now means is that developers can now spend more time implementing business logic rather than plumbing. This where real productivity gains can be made, as developers are usually domain experts and not infrastructure (transactions, security, threading and synchronization) experts. What's more, is that the developers don't need to learn a myriad of technologies to do this, because if they know .Net then they already have the lions share of the skills required.

There’s no steep learning curve for implementing infrastructure anymore, as these aspects are now out-of-the-box features that the developer gets for free. Well almost for free, as it’s now just a case of the developer changing their mindset about how they build their application. Once you understand that these infrastructure aspects are now taken care of for you, there’s no need to re-invent the wheel by trying to program them yourself anymore. And when it comes to advanced and complex things to implement like distributed transactions and synchronization, you really don’t want to be doing it yourself unless your name is Juval Lowy!

Summary

The DuraTrans project code is a complete system to demonstrate the how to build a fully transactional, durable data transfer system using WCF, MSMQ and Transactions.

Where possible I've tried to keep the code as simple as possible and I appreciate the fact that I could have used design patterns here and there. But I didn't want to distract from the core system and techniques, and so IOC, dependencey injection, factories and other good practices have been kept to a bare minimum. This is evident in the Media Service where I've kept to the basics for streaming the media, in reality chunking would be much better for larger files.

I've also left the business logic pretty bare too, just the essentials for the system to work. Any client specific business logic can simply be plugged in at the appropriate layers.

To see the durable side to this you can create 50 item to publish and start up the solution. Turn of one or all of the services and just let the client run. Then turn on or off the services. The system will keep processing, until all the items are published. If you keep the services off, you can see the messages being moved to the DLQ or the poisnon letter queue where they are handled appropriately. This is when manual intervention needs to happen as it really does mean there is a very real problem.

If all goes well, and all items are published, the publication service will displayed a message indicating the publication task has successfully been published. It's all very basic, and should give you enough information to see what's going on.

And lastly, each message displayed in the console apps also have the thread number displayed with it. This shows the system is mult-threaded, and yet there's no explicit thread code anywhere in the application.

The techniques and architecture principles used to design and build the DuraTrans system use the iDesign Method. http://www.idesign.net
I thoroughly recommend the WCF and Architects Master Classes taught by Juval Lowy if you want to learn more about software engineering and a ton of other stuff.

Setup

This is the first release, if you find a bug then use the issue tracker and I'll get things updated
You need to modify some of the app.config settings for the location of the media files and the database connection. You can find these seetings in the DAL.Publication project app.config and the BLL.MediaService project app.config.

You'll also need to create three private transactional message queues:

1. private$\PublicationQueue
2. private$\DistributionQueue
3. private$\publicationdlq

Use the ResetData stored procedure to reset the test data so you can run the publication again.

Future Developments

The current configuration of the DuraTrans project uses local message queues and basicHttpBinding for communication. I'll update this to use the iDesign queued HTTP bridge to allow the services of the DuraTrans system to run on geographically different servers. All the benefits of transactional message queues and all the benefits of a distributed architecture.

I'll implement the security features once the HTTP Bridge is put in place, everything is ready for it but I wanted to concentrate of the DuraTrans side of things rather than out-of-the-box WCF features.

Blog

Visit my blog to find out about other projects I'm currently working on. If anyone's interested in helping then get in touch!
http://www.outreal.com/wcf.html