Damian Hickey

Mostly software and .NET related. Mostly.

RavenDB and OWIN Webinar

Yesterday I did a webinar for RavenDB 3's Mystery Feature #2 - OWIN support. I'd like to thank Oren, Daniel and the HR team for giving me the opportunity to work on this. 

If you notice any typos, or things aren't as smooth as they should be, it's because it was my first webinar and I only starting preparing 45 mins before hand.

( It's tough to listen to my own voice :s )

Running RavenDB with OWIN and Katana

Just completed this spike and I'm happy with the outcome. Will post solution to Github soon.

Edit: Code available.

Upcoming changes to RavenDB nuget packages

Currently, as of last stable build #701, RavenDB is distributed in 3 NuGet packages:

  1. RavenDB-Client (contains client library, FSharp, Debugger Visualiser and MvcIntegration)
  2. RavenDB-Embedded (contains client library and embeddable database)
  3. RavenDB (contains client library and stand-alone server executable)

These are being transitioned to a more fine grained format with different package Ids that follow NuGet's package Id conventions, and will be available in build #731 (or soon there-after):

  1. RavenDB.Client (client library only, AsyncCtp dependency removed)
  2. RavenDB.Client.FSharp
  3. RavenDB.Client.Debug
  4. RavenDB.Client.MvcIntegration
  5. RavenDB.Database (useful for plugin / bundle developers)
  6. RavenDB.Embedded (depends on RavenDB.Client and RavenDB.Database instead of shipping with them)

With these packages you should be able to pick and choose precisely what you need and help reduce the number of dependencies attached to your projects.

The old packages will be continued to be produced and supported for the near future.

I hope to make bundle packages available in the coming weeks.

Any feedback is best sent to the RavenDB group.

RavenDB performance in your automated tests

In our automated testes, we usually have a single place that sets up a configured in-memory Raven database which might look something like this:

public static EmbeddableDocumentStore CreateDocumentStore(Assembly assembly)
{
	var store = new EmbeddableDocumentStore())
	store.UseEmbeddedHttpServer = false;
	store.RunInMemory = true;
	store.Initialize();
	IndexCreation.CreateIndexes(assembly, store);
	return store;
}

Over time, I noticed that our tests were geting slower and slower. Turned out that the IndexCreation process, which uses code generation underneath the hood, is a slow process that adds ~70 milliseconds per index per test. Multiply that by 10s of indexes and 1000s of tests and you've got quite a performance hit both on single test runs and test sessions (CI / NCrunch / Mightymoose / R# Test Session etc).

To improve the perfromance of this I removed "IndexCreation.CreateIndexes(assembly, store);" and made it such that we only put the exact index definitions that a test needs into the document store using a predicate. Thus minimising the document store creation time for an individul test.

But I think we can do better...

When running multiple tests in a session, we are often re-creating the same index over-and-over. Or rather, raven is emiting the same code, compiling and loading the same named type over-and-over. Adding a static Dictionary<string, Type> to the static Compile() method improved the performance of provisioning an embedable document store on second and subsequent test runs (if those tests used the same indexes) by at least a factor of 10. That is, second test (that used 2 indexes) run times went from >200ms to ~20ms. This also reduces the impact of a developer failing to properly restricted which indexes are created in a test.

(Side note: A positive side effect of this optimisation is reduced memory usage that is 'leaked' when dynamically loading assemblies.)

While there are probably going to be issues over my implementation hack of a compiled type cache (thread safety for one), I am going to try to get this improvement, or something similar, officially supported.

Update: Well that was quick. The optimization has been added to RavenDB and will be available in a build soon. Apparently it knocked 5-7 mins of Raven's build time. :)

Getting contextual state into a RavenDB's document store listener

Scenario: In multi-thread message driven application, I want to add contextual metadata to each and every modified and stored document. I would like to encapsulate this feature in such a way as so that it is done in one place and does not burden the event handlers with the responsibility.

An event handler may be supplied by seperate assembly and a contrived one that creates documents from events may look something like this:

public class MyEventHandler : IHandle<MyEvent>()
{
    private readonly IDocumentStore _store;
    
    public MyEventHandler(IDocumentStore store)
    {
        _store = store;
    }
    
    public void Handle(MyEvent @event)
    {
        using(var session = _store.OpenSession())
        {
            var doc = new MyDocument { // Populate properties from message };
            session.Store(doc);
            session.SaveChanges();
        }
    }
}

This handler is typically invoked at the point where the message enters this system in the main application and then

public void OnMessage<T>(Message<T> message)
{
    var handlers = _handlerResolver.ResolveAll<T>();
    foreach(var handler in handlers)
    {
        handler.Handle(message.Body);
    }
}

This method is reentrant and may (will) be called from more than one thread as the same time.

I would like to add contextual information, here message.Id, to each document that is created or modified as a result handling this message. Do do so I need to intercept the document etity instance just before it is saved. This is achieved through implementing the IDocumentStoreListener interface which looks like...

public interface IDocumentStoreListener
{
    void AfterStore(string key, object entityInstance, RavenJObject metadata);
    bool BeforeStore(string key, object entityInstance, RavenJObject metadata);
}

DocumentStoreListeners are registered before initializing the document store and like the DocumentStore, the listener's lifecyle is singleton with respect to the main application. Therefore it must be reentrant and thread safe.

var store = new DocumentStore { // Uri = etc };
store.RegisterListener(new EventIdDocumentStoreListener);
store.Initialize();

...where my implementation may look something like this (just BeforeStore() implemention shown for berevity):

public class EventIdDocumentStoreListener : IDocumentStoreListener
{
    public const string PropertyName = "LastMessageId";
    
    public bool BeforeStore(string key, object entityInstance, RavenJObject metadata)
    {
        RavenJToken token = RavenJToken.FromObject(messageId);
        if (metadata[PropertyName] == null)
        {
            metadata.Add(PropertyName, token);
        }
        else
        {
            metadata[PropertyName] = token;
        }
        return false;
    }
}

See the problem here? I have no obvious mechanism to supply the messageId into this method from which the RavenJToken is then constructed from. The solution I have arrived at is to use thread local storage, specifically .NET4's ThreadLocal<T>. This will allow me to store messageId value in a place that is specific to the running thread. I set it in the OnMessage dispatcher and pick it up in the listener. This ThreadLocal instance is also singleton with respect to the main application.

Now our listener will depend on a ThreadLocal:

public class EventIdDocumentStoreListener : IDocumentStoreListener
{
    public const string PropertyName = "LastMessageId";
    private readonly ThreadLocal<Guid> messageIdThreadLocal;
    
    public EventIdDocumentStoreListener(ThreadLocal<Guid> messageIdThreadLocal)
    {
        _messageIdThreadLocal = messageIdThreadLocal;
    }
    
    bool BeforeStore(string key, object entityInstance, RavenJObject metadata)
    {
            RavenJToken token = RavenJToken.FromObject(_messageIdThreadLocal.Value);
            if (metaData["LastMessageId"] == null)
            {
                metaData.Add(PropertyName, token);
            }
            else
            {
                metaData["LastMessageId"] = token;
            }
        }
    }
}

And our OnMessage dispatcher can set the message ID for the specific thread before calling the handlers:

public void OnMessage<T>(Message<T> message)
{
    var handlers = _handlerResolver.ResolveAll<T>();
    _messageIdThreadLocal.Value = message.Id;
    foreach(var handler in handlers)
    {
        handler.Handle(message.Body);
    }
}

So that is how I supply contextual state to a listeners in a safe and transparent manner.

Notes
- This code above is to demonstrate using ThreadLocal to pass contextual state into a RavenDB IDocumentStoreListener. It does not represent a production quality solution :)
- This solution does not work in an asp.net / wcf environment. Modifying above to do that is left as an exercise to the reader (hint: HttpContext.Items)