Damian Hickey

Mostly software and .NET related. Mostly.

Upcoming changes to RavenDB nuget packages

Currently, as of last stable build #701, RavenDB is distributed in 3 NuGet packages:

  1. RavenDB-Client (contains client library, FSharp, Debugger Visualiser and MvcIntegration)
  2. RavenDB-Embedded (contains client library and embeddable database)
  3. RavenDB (contains client library and stand-alone server executable)

These are being transitioned to a more fine grained format with different package Ids that follow NuGet's package Id conventions, and will be available in build #731 (or soon there-after):

  1. RavenDB.Client (client library only, AsyncCtp dependency removed)
  2. RavenDB.Client.FSharp
  3. RavenDB.Client.Debug
  4. RavenDB.Client.MvcIntegration
  5. RavenDB.Database (useful for plugin / bundle developers)
  6. RavenDB.Embedded (depends on RavenDB.Client and RavenDB.Database instead of shipping with them)

With these packages you should be able to pick and choose precisely what you need and help reduce the number of dependencies attached to your projects.

The old packages will be continued to be produced and supported for the near future.

I hope to make bundle packages available in the coming weeks.

Any feedback is best sent to the RavenDB group.

RavenDB NuGet Packages: Coarse grained vs fine grained and sub-dependencies

This topic has recently come up on the RavenDB list (1, 2) and Ayende’s blog. I've been down that road of fine-grained packages (internal code) and back again so this is my opinion based on recent experience. The current opinion of the RavenDB team is that they want to avoid having 10s of nuget packages.

So, is 10’sof nuget packages really a problem and if so, for whom and how?

The Package Consumer

From the package consumer side, fine grained packages allows them to pick and choose precisely what they want without crufting up their project's references with unnecessary extraneous references. (Unnecessary references are a pet hate of mine). There are a number of OSS projects that appear to be doing this successfully such as Ninject , ServiceStack and NServiceBus.

One of the consumer’s concerns is that if they do pick 2 packages where one is dependent on the other, is that of package versioning and updating. If they were to pick RavenDB-Client and (a hypothetical) RavenDB-Client(Debug), they expect that at the precise moment one package is updated, then the other one is done so too, such that updating a solution is easy. That is unless RavenDB team is exercising flawless semantic versioning, which I doubt.

The other concern, regardless of a coarse-grained or fined grained packaging strategy, is that of package sub-dependencies. Despite the best intentions of authors with semver and specifying package version ranges in nuspec files , this is an area of brittleness, as was recently demonstrated by a recent log4net package update. Also specifying a package dependency because your package uses it internally unfairly makes your consumer dependent on it. Introduce another package that has the same dependency but maybe a different version ant they are at risk of runtime exceptions, deployment conflicts and having to perform brittle assembly redirect hacks.

Currently, adding a RavenDB-Client package to a class library adds the following 8 references:

image

… and the following package dependencies:

image

My fresh class library is now dependent on a specific Logging Framework, some MVC stuff that has nothing to do with what I am doing and a Community Technology Preview library that may or may not have redistribution licence concerns. This isn’t a great experience. A brief analysis:

  1. AsyncCtpLibrary’s usage is entirely internal to Raven.Client.Lightweight could be ilmerged and internalized. Example projects that do this approach include Automapper and Moq.
  2. Newtonsoft.Json is exposed through Raven.Client.Lightweight’s exported types so is a real dependency.
  3. NLog? There is a better way.
  4. Raven.Abstractions appears to contain both client side and server side concerns. The client side ones could be extracted and placed into Raven.Client.Lightweight and referenced by server side assemblies. (Perhaps, don’t know enough to say for sure)
  5. Raven.Client.MvcIntegration and .Debug are entirely optional and could be provided by separate packages, if I wanted them.
  6. System.ComponentModel.Composition could probably be removed if the server side abstractions were not part of the client package.

The end result, in my opinion, should look like this:

image

If the concerns of minimizing sub package dependencies and lock-stepping of package releases are addressed, then I believe that fine-grained packages are desirable to a package consumer.

Package Producer

The primary concern on the producer side is one of maintenance. Adding additional fine-grained package specifications to a single solution does have a cost, but I’d argue that it’s worth it when considering benefits to the consumer.

Where things do get difficult fast for the producer though is when the fine grain packages are independently versioned. Previously I said I doubted Raven is doing flawless semantic versioning. I doubt anyone is doing it flawlessly because there is no tooling available to enforce and you can’t rely on humans. I’ve tried the automatic CI based package updating “ripple” where Solution B that produces Package B but depends on Package A from solution A , automatically updates itself when a new version of Package A is produced. It didn’t work reliably at all. If the producer has a lot of fine-grained solutions and they have a lot of momentum, package management quickly becomes a mess and a massive time sink.

But if the package producer is using a single solution (as is the case of RavenDB) and a concurrent release of all fine-grained packages at the same time is performed, the cost of supporting fine grained packages is not prohibitive. This is the approach currently taken by ServiceStack.

Simple Powershell script to tidy NuGet's packages.config

This:

param([string]$path = ".")

$packagesConfigFiles = Get-ChildItem $path -Recurse | Where-Object {$_.Name -eq "packages.config"}

if($packagesConfigFiles -eq $null){
	Write-Error "No packages found"
	return
}

foreach($packagesConfig in $packagesConfigFiles){
	Write-Host "Tidying" $packagesConfig.FullName
	$xml = [xml](Get-Content $packagesConfig.FullName)
	$sortedNodes = $xml.packages | select-xml 'package' | Select-Object -expand Node | sort 'id'

	[xml]$sortedXml = "<?xml version='1.0' encoding='utf-8'?>
	<packages>
		$($sortedNodes | % { $_.OuterXml})
	</packages>"

	$sortedXml.Save($packagesConfig.FullName)
}

...will turn this:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Iesi.Collections" version="3.1.0.4000" />
  <package id="NHibernate" version="3.1.0.4000" />
  <package id="Castle.Core" version="2.5.2" />
  <package id="NHibernate.Castle" version="3.1.0.4000" />
  <package id="FluentNHibernate" version="1.2.0.712" />
  <package id="Microsoft.IdentityModel" version="6.1.7600" />
  <package id="NBuilder" version="2.3.0.0" />
  <package id="FluentMigrator" version="0.9.0.1" />
  <package id="Ninject" version="2.2.1.0" />
  <package id="Ninject.Extensions.Wcf" version="2.2.0.0" />
  <package id="Ninject.Web.Mvc2" version="2.2.0.1" />
  <package id="Common.Logging" version="1.2.0" />
  <package id="Rx-Core" version="1.0.2856.0" />
  <package id="Compare-NET-Objects" version="1.0.2.0" />
  <package id="StoryQ" version="2.0.5" />
  <package id="Moq" version="4.0.10827" />
  <package id="xunit" version="1.8.0.1545" />
  <package id="xunit.extensions" version="1.8.0.1545" />
</packages>

...into this:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Castle.Core" version="2.5.2" />
  <package id="Common.Logging" version="1.2.0" />
  <package id="Compare-NET-Objects" version="1.0.2.0" />
  <package id="FluentMigrator" version="0.9.0.1" />
  <package id="FluentNHibernate" version="1.2.0.712" />
  <package id="Iesi.Collections" version="3.1.0.4000" />
  <package id="Microsoft.IdentityModel" version="6.1.7600" />
  <package id="Moq" version="4.0.10827" />
  <package id="NBuilder" version="2.3.0.0" />
  <package id="NHibernate" version="3.1.0.4000" />
  <package id="NHibernate.Castle" version="3.1.0.4000" />
  <package id="Ninject" version="2.2.1.0" />
  <package id="Ninject.Extensions.Wcf" version="2.2.0.0" />
  <package id="Ninject.Web.Mvc2" version="2.2.0.1" />
  <package id="Rx-Core" version="1.0.2856.0" />
  <package id="StoryQ" version="2.0.5" />
  <package id="xunit" version="1.8.0.1545" />
  <package id="xunit.extensions" version="1.8.0.1545" />
</packages>

To be honest, I'd prefer if NuGet did this itself.

A NuGet (for tools) / OpenWrap (for libs) hybrid Visual Studio solution template

I've created a solution template as a that may form the basis of reference solution to be used in my company. It's available to view here

Requirements & Design decisions:

  1. To be used in a corp environment.
  2. No packages / wraps / dependencies / tools are committed to version control, except the bare minimum to bootstrap the build and/or restore dependencies.
  3. Packages / wraps / dependencies / tools may come from a variety of sources - nuget galley, local nuget repo, local openwrap repo.
  4. Anyone should be able to download the source and single click build the solution with no additional effort, friction or setup. (i.e. just run Build.ps1 )
  5. Easily managed via CI without addition setup and minimal coupling (i.e. .\Build.ps1 -buildNumber 2 )
  6. Sticking with released version of tools.
  7. I am using NuGet as my 'tools' manager. OpenWrap does not appear to be suitable for this purpose.
  8. OpenWrap seems to want to be responsible for compiling, running tests and creating the package. I already have psake for that. In my mind, creating a packages is a post-build step. Anyway, build: none to the rescue. 
  9. A psake build task sets the version number (using a build number if supplied) and keeps SharedAssemblyInfo.cs and the version file in sync.
  10. We use code contracts and the contract assemblies must be shipped in the wrap.

Some of the things I don't like:

  1. There is a bit too much going on in the root of the project, but at least Build.ps1 is at the top.
  2. Would rather not have to commit warps\openwrap-1.0.0.53270092.wrap to  support the bootstrap.
  3. ... and I'd like to add wraps/* to .hgignore.
  4. Can't isolate XUnit to only the DH.SolutionTemplate.Tests project (I know something is in the pipeline for this).
  5. Explicitly defining what goes into the resulting wrap may be a place where mistakes can easily happen. Not sure how to mitigate that.
  6. Having to use ;-shellInstall none; everywhere. I think o.exe default behaviour should be 'do nothing' and installation is an explicit step. I don't know enough of OpenWraps's architecture or intention to hold that belief firmly though :)
  7. In RestoreDependencies.ps1 I am deliberately adding the NuGet gallery remote source. This will add it permanently to the users remote list which I'd prefer not to affect. (I know there is something in OpenWrap's pipeline for this).

So, if there are any places this could be improved, let me know.

PowerShell script to restore a project's packages and tools

Requires nuget.exe to be in the solution root and commited to source control.

# Tools
.\NuGet.exe i DotCover -s \\myserver\Dev\NuGetPackages -o Tools
.\NuGet.exe i StyleCopCmd -s \\myserver\Dev\NuGetPackages -o Tools

# Dependencies
$packageConfigs = Get-ChildItem . -Recurse | where{$_.Name -eq "packages.config"}
foreach($packageConfig in $packageConfigs){
  Write-Host "Restoring" $packageConfig.FullName
  .\nuget.exe i $packageConfig.FullName -o Source\Packages
}