Damian Hickey

Mostly software and .NET related. Mostly.

How I am using OWIN

Using OWIN has enabled me to build an application server with the following features:

  • Self-hosted web apps and services using various frameworks
  • Unified authentication mechanism
  • Ability to DI my application's services into Fubu controllers, ServiceStack services, SignalR PersistentConnections etc
  • Simpler end-to-end acceptance tests without having to go through the network stack
  • All assets (html, css, js, etc) are embedded resources
  • A delightfully simple deployment - a single .exe

I like this..  a lot! Nice job OWIN crew and supporting framework devs.

It is not possible to do this with asp.net due to it's tight coupling with IIS, but it looks like it will support OWIN in the future.

* Just a note on the diagram - RavenDB doesn't support owin yet, but I intend to tackle it soon.

Update: I'm in the process of reducing the number of components in this, specifically replacing the FubuMVC component with NancyFX. Mostly because of maintainance and future development and not anything owin specific.

Securing ServiceStack using Azure Authentication Library and WPF Client

I've pushed to github a port of the AAL - Native Application to REST service - Authentication via Browser Dialog sample code that, instead of using WebAPI, uses Service Stack.

My implementation uses ServiceStack's request filters to verify that all requests have a valid security token in a similar way the sample WebAPI uses a delegating http handler. My understanding of ServiceStack's Authentication and authorization feature, as is currently designed, is that it doesn't support the situation where you authenticate externally first; ServiceStack wants to authenticate on your behalf through one of it's auth services.

As this is my first time playing with both ServiceStacks authentication and Azure Authentication Library, I'm sure there are things that can be improved. If so, let me know here, github or twitter :)

Up next - securing a Nancy app and any OWIN app with AAL.

Note: the original sample did not have any tests, unit or otherwise, so I just F5'd my way through this against my better judgement.

ServiceStack OWIN support sneak peek

Something I am currently kicking the tyres of, and am sucessfully using in a limited fashion thus far, is using ServiceStack on OWIN. The code currently resides on a branch in my fork and I'm going to bake it a bit more before sending a pull request. 

So what will it look like?

Using ServiceStack's HelloWorld as an example, we create the OWIN Host:

public class HelloOwinHost : OwinHostBase
{
	public HelloOwinHost() : base("Hello Web Services", typeof(HelloService).Assembly) { }

	public override void Configure(Container container)
	{
		Routes
		  .Add<Hello>("/hello")
		  .Add<Hello>("/hello/{Name}");
	}
}

Only difference between this and the original example is that we're inheriting from OwinHostBase instead of AppHostBase.

Here is how you host this on Microsoft.HttpListener.Owin - a server based on .net http listener from the KatanaProject - and using some helper nuget packages (Owin.Startup and Owin.Builder):

//Initialize the Owin host
var helloOwinHost = new HelloOwinHost();
helloOwinHost.Init();

// Use Owin.Startup.AppBuilder to help configure the server
var app = new AppBuilder();
ServerFactory.Initialize(app.Properties);

//We tell the server where to listen and where to delegate requests to
app.SetHostAddress("http", "+", 8080)
	.Run(new Func<IDictionary<string, object>, Task>(helloOwinHost.ProcessRequest));

//Create the server and test with a client
using (ServerFactory.Create(app.Build(), app.Properties))
using(var client = new XmlServiceClient("http://localhost:8080"))
{
	var response = client.Send<HelloResponse>(new Hello {Name = "Damian"});
	Assert.AreEqual("Hello, Damian", response.Result);
}

Overall, pretty standard OWIN stuff. Full source of this example is on github if you want to clone and run it.

The interesting things start to happen when you combine multiple OWIN supporting frameworks (Nancy, Signalr, ServiceStack) behind a single server in your own process... more on that later :)

Upcoming improvements in RavenDB's packages

As of the current release, 1.0.960, when installing RavenDB.Embedded your project's references would look something like this, not including your own references:

For 1.2.x, it looks like this:

This pleases me.

The main changes include:

  1. Raven now uses a dynamic logging abstraction. The RavenDB.Client and .Database projects no longer directly depend on NLog, but actually support NLog and Log4Net dynamically. All you have to do is reference either library and RavenDB will use it. The logging http endpoint is now per database.
  2. Newtonsoft.Json, a source of much grief amonst the current 1.0.x users, has been imported and internalized. 
  3. Raven.Munin, Raven.Storage.Esent and Raven.Storage.Managed have been merged into Raven.Database.
  4. Raven.Backup and Raven.Smuggler... well they shouldn't have been there in the first place.
  5. The packages use the <references> element to explicitly define what should be referenced, so BouncyCastle.Crypto, Lucene.* etc will no longer be exposed to you.
  6. Raven.Studio.xap is embedded into Raven.Database, so it will no longer be added to your project. You can still deploy an updated xap which will override the embedded one, if you so wish.
  7. Raven.Client.Debug has been discontinued.

You can try these out in the next day or so by installing the unstable packages from nuget ( install-package RavenDB.Embedded -pre). Any feedback, please post to the google group.

Testing EventStore storage engines, part 2 - MongoDB

A follower on twitter, @red_square, ran my tests from my previous blog post using MongoDB as a storage engine that gave some interesting event read rates.

I incorporated this into my test:

Compared to the RDBMs, these reads speads are blistering: 7 times faster than SQL Server and over 20 times faster than PostgreSQL. There is a cost to this, and we are comparing apples and oranges here. There is plenty on the web comparing MongoDB and other NoSql storage tech to the traditional so I'm not going to regurgitate them here. 

For my case, I think I'd like to still present the option of using an RDMBS for our customers. I think they are just more comfortable with it. But it does open the door to an interesing optimization strategy. There are cases, such as deploying new projection schema, upgrading an exisiting one, recovering from a crash, or replaying to point-in-time for analysis, where you want a pipeline that can rebuild the projections as quickly as possible. So perhaps using MongoDB as a secondary event store (with lower reliabilty requirements) which gives you a super fast read pipeline could be feasible. This secondary store can be a local async mirror of the primary store that will get you 99% of the required events for a projection store re-build (it may not have 100% of events due to it being an async mirror). The remaining events can the come from the slower primary event store at the end of the rebuild step. If the MongoDB mirror is ever lost, such as in disaster recovery, it can just be re-mirrored. I'd hope this would be a very rare occurance

(Big thanks to Steve Flitcroft for the MongoDB help)

Edit: The specific numbers are not representative of real world numbers. The interesting bit is the relative difference between them, ceteris paribus.

Testing EventStore storage engines

From the project's home page: "The EventStore is a persistence library used to abstract different storage implementations when using event sourcing as storage mechanism." From the same page, you can see it either currently or will in the future, support a myriad of storage engines.

But which one performs the best?

My typical usage scenario is:

  1. Handle commands in parallel, where each command generally results in a single event being appended to a single stream.
  2. Read all / many events when I need to fully / partially rebuild a projection.

I've upload a project to github the test 3 engines. (I've only just hacked this up, so any and all feedback is welcome). The test isn't meant to be very scientific, there are a lot factors to consider. At this point I'm really only interested in getting a feeling for the relative differences between each engine.

Test server details:

  1. Windows 2008 R2 VM running in Virtual Box.
  2. 8GB RAM
  3. 4 CPU @ 4.3Ghz
  4. HD the VDI is on is a 750GB WD SATA3.
  5. SqlServer 2008R2 SP1, PostgreSql(9.1), and MySql(5.2) installed and running as services. Installations were clean and settings were left as default where possible.
  6. Comms between dev machine and server via BridgedAdapter.

Each test is run 3 times.

Results

Running test app on test server:

| DB                 | Run # | Stream Count | Events/Stream | Total Events | Insert Time (s) | Insert Rate (events/s) | Read Rate (events/s) |
| SqlServerPerfTest  | 0     | 100          | 100           | 10000        | 18.9264927      | 528                    | 10955                |
| SqlServerPerfTest  | 1     | 100          | 100           | 10000        | 19.2425916      | 520                    | 10655                |
| SqlServerPerfTest  | 2     | 100          | 100           | 10000        | 20.0126372      | 500                    | 12518                |
| PostgreSqlPerfTest | 0     | 100          | 100           | 10000        | 82.9118079      | 121                    | 4366                 |
| PostgreSqlPerfTest | 1     | 100          | 100           | 10000        | 79.6413285      | 126                    | 4396                 |
| PostgreSqlPerfTest | 2     | 100          | 100           | 10000        | 79.3978265      | 126                    | 4383                 |
| MySqlPerfTest      | 0     | 100          | 100           | 10000        | 33.5853395      | 298                    | 9633                 |
| MySqlPerfTest      | 1     | 100          | 100           | 10000        | 33.4475381      | 299                    | 10340                |
| MySqlPerfTest      | 2     | 100          | 100           | 10000        | 27.4845492      | 364                    | 12302                |

Running the app from dev machine, communicating via BridgedAdapter:

| DB                 | Run # | Stream Count | Events/Stream | Total Events | Insert Time (s) | Insert Rate (events/s) | Read Rate (events/s) |
| SqlServerPerfTest  | 0     | 100          | 100           | 10000        | 14.8824588      | 672                    | 14748                |
| SqlServerPerfTest  | 1     | 100          | 100           | 10000        | 16.6571154      | 600                    | 14805                |
| SqlServerPerfTest  | 2     | 100          | 100           | 10000        | 17.720558       | 564                    | 14749                |
| PostgreSqlPerfTest | 0     | 100          | 100           | 10000        | 69.212226       | 144                    | 4643                 |
| PostgreSqlPerfTest | 1     | 100          | 100           | 10000        | 72.2258403      | 138                    | 4647                 |
| PostgreSqlPerfTest | 2     | 100          | 100           | 10000        | 73.3332329      | 136                    | 4633                 |
| MySqlPerfTest      | 0     | 100          | 100           | 10000        | 34.1016615      | 293                    | 14624                |
| MySqlPerfTest      | 1     | 100          | 100           | 10000        | 34.7092424      | 288                    | 14585                |
| MySqlPerfTest      | 2     | 100          | 100           | 10000        | 34.4671897      | 290                    | 14595                |

Finally, running from laptop over Gb LAN:

| DB                 | Run # | Stream Count | Events/Stream | Total Events | Insert Time (s) | Insert Rate (events/s) | Read Rate (events/s) |
| SqlServerPerfTest  | 0     | 100          | 100           | 10000        | 27.4017972      | 365                    | 4293                 |
| SqlServerPerfTest  | 1     | 100          | 100           | 10000        | 27.5854037      | 363                    | 5282                 |
| SqlServerPerfTest  | 2     | 100          | 100           | 10000        | 30.104196       | 332                    | 5417                 |
| PostgreSqlPerfTest | 0     | 100          | 100           | 10000        | 133.9792362     | 75                     | 1942                 |
| PostgreSqlPerfTest | 1     | 100          | 100           | 10000        | 145.4393013     | 69                     | 2654                 |
| PostgreSqlPerfTest | 2     | 100          | 100           | 10000        | 154.967134      | 65                     | 2442                 |
| MySqlPerfTest      | 0     | 100          | 100           | 10000        | 44.8102286      | 223                    | 4756                 |
| MySqlPerfTest      | 1     | 100          | 100           | 10000        | 44.735568       | 224                    | 4837                 |
| MySqlPerfTest      | 2     | 100          | 100           | 10000        | 45.8796399      | 218                    | 4540                 |

The laptop is no where near as powerful as the desktop. During the exectution of the test, it's CPU was pegging around 90-95 percent which was a limiting factor. I expected that and the network I/O to give lower numbers, which it did. I also expected the numbers to level out a bit as the bottle neck can no longer be completely attributed to be the database. But that didn't happen - PostgreSql was slower by the same factor as when running the test app on the latptop as on the server itself. This leads me to suspect the npgsql data provider is not up to scratch.

Anyway, these are just preliminary numbers, and I'm sure there are lots of holes here. They're nonetheless intesting.

Our package, build and deployment pipeline (NuGet, MyGet, ProGet, TeamCity and OctopusDeploy Heaven)

I don't have this fully baked yet, some of the software is barely V1 and there are still some kinks to work out, but I am finding this relatively easy to manage. Compared to a couple of years ago, when there was virtually nothing like this in the .net space (MSIs do not count).

How we are doing Domain Event versioning

(x/post and edited from the DDD/CQRS group)

Edit: Some background info... A Domain Event is expected to be around forever. This means that we need to be able to read and deserialize events that could be 5, 7 or a decade old. Think of them as little file schemas - we need to keep the knowledge of that schema and keep it intact the lifetime our the data, which may be longer than the application itself.

So, I think this will work for BEAM for and our particular requirements - I'm certainly not prescribing this as 'best practice' or anything. I've decided to have two representations of a Domain Event, which I have called "Application Domain Event" and the other "Storage Domain Event".

Application Domain Event:

  1. Is published on bus.
  2. Can be referenced and depended on by any part of the application / system.
  3. Can follow OO patterns, such as DRY, i.e. sharing a type / enum with a command in a common location is permitted.
  4. Is versioned is the standard .net application / assembly manner. Old versions are not kept around.

Storage Domain Event:

  1. Is the schema format for serializing to, and deserializing from, the event store.
  2. Is in a project that has no external dependencies, except for 'System'. (And maybe System.Runtime.Serialization if they need to be marked [DataContract] etc).
  3. Is not referenced or consumed by any part of the system / application except for the event store.
  4. Versioning is done for the entire set of domain events, rather than individual. Currently this is organised by namespace, but may do seperate assemblies later.
  5. Will be kept around 'forever'.
  6. Will, on occasion, utilize wholesale migration if it makes sense.

I use Jonathan Oliver's event store library and am utilizing pipeline hooks to convert between Storage and Application events when read (up conversion), when committed (commit conversion) and published (an IDispatchCommits wrapper). These converters are in a separate library and are all, at this point, AutoMapper based with minimal configuration. There is now an additional developer cost in creating / managing the storage representation as well as the application one, but it doesn't seem to be completely wild (yet). A test using AutoMapper's AssertConfigurationIsValid catches most event shape mismatch issues very quickly. There is also a perf hit in the mapping but my gut tells me it's small compared to serialization and event store I/O.

For 'locking' a version of the storage events I have a glorified copy -> fine/replace script+tool (me-ware) that duplicates a storage version, changes the namespace (i.e. ".V1." -> ".V2."), generates a type signature, creates a test to make sure that types don't change and updates converters. I'm not convinced that this is ultimately necessary or worth it. I have some time to chew on that and am in no particular rush to settle on an approach.

So, will this stand the test of time? Time will tell I suppose :)

RavenDB Revisions Bundle

While Raven's 'Versioning' bundle is designed for regulatad environments, such as healtcare, where nothing can be deleted, this bundle is more useful for applications where the document is a result of a projection (i.e. CQRS), where the application controls the document version numbers. It differs in the following ways:

    It only works on documents that have a Revision property.
    It allows Revisions to be deleted.
    It allows Revisions to be overwritten.
    Your application controls the Revision number and not the plugin.
    Your application is resposible for revision number contiguity.


Install

Packages are available nuget.org (source on github):

Using

If you want your this plugin to automatically create revision copies add a Revision property to your document:

public class MyDocument
{
    public string Id { get; set;}
   
    public int Revision { get; set; }
}

When a revisionable document is saved, the plugin will create a copy with a new id. These copies are excluded from all indexes. You should be aware that these revisions are kept indefinitely until you have specifically deleted them or used Raven's 'Expiration' bundle to perform automatic purging.

To retireve a specific revision of a document use the LoadRevision extension method:

using(var session = store.OpenSession)
{
    var doc = session.LoadRevision("key", 2);
}

To delete a specific revision of a document:

_documentStore.DatabaseCommands.DeleteRevision("key", 2, null);

 

RavenDB Client Contextual Listeners

In a previous post, I described how to get contextual information into a RavenDB client listener. I've since create a NuGet package (source) to make things a little easier.

To recap, listeners allow you to hook into all document operations that occur in each session. The current design of the listeners is that their lifecyle is singleton with respect to the document database, making it a little tricky to pass in contextual information.

There are 4 listeners interfaces define in Raven.Abstractions that allow you to intercept the data sent to, or recieved from, a Raven database:

    IDocumentStoreListener
    IDocumentDeleteListener
    IDocumentConversionListener
    IDocumentQueryListener

So correspondingly, there are 4 context aware listeners that you can register:

    ContextualDocumentStoreListener<>
    ContextualDocumentDeleteListener<>
    ContextualDocumentConversionListener<>
    ContextualDocumentQueryListener<>

... and there are 4 base listener context types that you inherit from that perform the contextual work:

    AbstractDocumentStoreListenerContext
    AbstractDocumentDeleteListenerContext
    AbstractDocumentConversionListenerContext
    AbstractDocumentQueryListenerContext

An example is the best way to demonstrate how to use these.

Example

Say you want to add the current username to the metadata of each document that is stored. Therefore we want an IDocumentStoreListener to intercept each document before storing and add our username metadata.

First define your username document store context type:

public class UserNameContext : AbstractDocumentStoreListenerContext
{
    private readonly string _userName;

    public UserNameContext(string userName)
    {
        _userName = userName;
    }

    protected override void AfterStore(string key, object entityInstance, RavenJObject metadata)
    {}

    protected override bool BeforeStore(string key, object entityInstance, RavenJObject metadata)
    {
        metadata.Add("UserName", RavenJToken.FromObject(_userName));
        return false; //return true if you modify the entityInstance
    }
}

Register your the listener that will be aware of the UserNameContext:

var documentStore = new DocumentStore()
{
    Url = "http://server"
};

// can be called after .Initialize(), doesn't matter
documentStore.RegisterListener(new ContextualDocumentStoreListener<UserNameContext>());
documentStore.Initialize();

And the sweet bit.. just open a new context before you open a session:

using(new UserNameContext("The UserName"))
using(var session = documentStore.OpenSession())
{
    session.Store(new Doc());
    session.SaveChanges();
}