ORMBattle.NET

The next update of our test suite is almost ready.

Changes in LINQ tests:

• LINQ tests are categorized now, although initially scores for each category will be available only in Output.txt.
• There are about 130 tests now (instead of agruable 100), so we calculate both % and actual count of passed and failed tests.
• Percentage of complex LINQ tests (tests that untilize several different features) is reduced.

Changes in performance tests:

• There are *Single test sequences: InsertSingle, UpdateSingle and RemoveSingle. They show performance of CUD operations when framework can't batch them by some reason.
• We run tests for 50, 100, 500, 1000, 5000, 10000 and 30000 item sequences.

Changes in code browser:

• A bug with wrong indent (1 instead of 2 for our files) is fixed.
• It automatically opens file tree branch on navigation by URL.
• There are other minor changes, e.g. now it shows .bat files.

We're planning to update our scorecard and upload new code to code.ormbattle.net on this weekend.

"Sure, there have been mistakes on both side, but I definitely agree with the necessity of having a good and non-biased ORM testing site. Can you even imagine buying a video card, CPU, or HDD without having a bunch of benchmarks to look at?! Imagine the reaction if AMD/ATI or NVIDIA would ask to be removed from a benchmarking test. They'd be laughed out of the room!"

There is simply nothing to add.

Frankly speaking, I'm tired to protect the results you see on this web site during last 1.5 weeks. But since I wrote "Part 1" in my previous post, I have no choice now :) Although I hope, this is the last part.

So first of all, some links:

• ADO.NET Entity Framework Performance Comparison. It is based on two posts: first one is about materialization performance, and the second one is about query performance. Do you see the parallels?
• ADO.NET performance improvements with the .NET Framework 2.0 SP1 - a bit old, and about ADO.NET itself, but take a look on tests they run.
• Batching support in NHibernate - "As of about 90 minutes ago, NHibernate has batching support. :-D", and you know who is the author :-D Of course, there are other posts in his blog related to NH performance. E.g. this one, that refers to that one, made me smile ;)
• Subsonic scaling - "Test Number One: Inserting Lots Of Data", "Test Number Two: Pulling Records Back Out", "Test Number Three: Loading Collections", "Test Number Three: Loading Bigger Collections – 100 at a Time". Rob, why do you dislike our benchmark then? It looks like we prefer to test the same logic...

Ok, I hope that's enough. The list is short because I googled it in just 10 minutes.

And finally, thoughts:

I got a lot of critics. A lot! And most part of it was based on the fact this benchmark is biased, because we are ORM vendors. But take a look at this reply. I think, it's great.

Think about the following: are the people criticizing ORMBattle.NET most (Oren, Frans, Rob, Fabio) really represent independent and non biased point of view? No. All of them are ORM developers. And it does not really matter, if their product is free or not. Thus I don't see any reason to consider their critics even a bit more "independent" than results you see here.

As many of you know, Oren Eini (likely, the most famous NHibernate developer) has made two posts related to this test suite. Here they are: "Benchmarks are useless, yes, again" and "Benchmark cheating, a how to" (btw, I still think it was not cheating - we've fixed everything almost immediately, so is was simply a mistake). Anyway, I posted tons of arguments and facts to Oren's blog, but in general, I got feeling they were not accepted.

So I was thinking, is there any simple proof that Oren actually said something different, rather than truth... I clearly see I can't make him confirm he was at least partially wrong. On the other hand, in fact, he made a statement that performance of NHibernate is already good enough, and there is really nothing to optimize. Can I get something out of this? I think, yes:

1. We've shown our product wins almost all the competitors on CUD tests. I confirm this is achieved mainly because of automatic batching of CUD sequences. It allows us to outperform frameworks without this feature by 2-3 times on CUD tests. So I expect I will never see this feature implemented in NHibernate, since, as Oren claims, NHibernate is already good enough.

2. Three frameworks shown here have exposed outstanding results on materialization test. Currently (i.e. after all the fixes) NHibernate is 8-10 times slower than EF, Lightspeed and our own product on this test. So I expect I will never see NHibernate materialization performance is getting higher than e.g. 20% of the best one between these 3 tools. By the same reason.

Since we track performance on our tests here, we'll immediately see, if any of these conditions will be violated. And if this ever happen, I expect Oren must at least publicly apologize for his exceptional rhino obstinacy ;)

And... Since Frans Bouma has been supporting him exceptionally well there, I think he must accept the same conditions as well.

Obviously, later I'll provide other evidences in addition to this simple game. But I'd like to publish this first.

P.S. Guys, nothing personal. I love reading your blogs and I know you're quite clever persons. But I think if you're standing on your point of view so hard, you are ready to prove it in action. Quite simple.

Kind regards, Alex Yakunin