As many of you know, Oren Eini (likely, the most famous NHibernate developer) has made two posts related to this test suite. Here they are: "Benchmarks are useless, yes, again" and "Benchmark cheating, a how to" (btw, I still think it was not cheating - we've fixed everything almost immediately, so is was simply a mistake). Anyway, I posted tons of arguments and facts to Oren's blog, but in general, I got feeling they were not accepted.
So I was thinking, is there any simple proof that Oren actually said something different, rather than truth... I clearly see I can't make him confirm he was at least partially wrong. On the other hand, in fact, he made a statement that performance of NHibernate is already good enough, and there is really nothing to optimize. Can I get something out of this? I think, yes:
1. We've shown our product wins almost all the competitors on CUD tests. I confirm this is achieved mainly because of automatic batching of CUD sequences. It allows us to outperform frameworks without this feature by 2-3 times on CUD tests. So I expect I will never see this feature implemented in NHibernate, since, as Oren claims, NHibernate is already good enough.
2. Three frameworks shown here have exposed outstanding results on materialization test. Currently (i.e. after all the fixes) NHibernate is 8-10 times slower than EF, Lightspeed and our own product on this test. So I expect I will never see NHibernate materialization performance is getting higher than e.g. 20% of the best one between these 3 tools. By the same reason.
Since we track performance on our tests here, we'll immediately see, if any of these conditions will be violated. And if this ever happen, I expect Oren must at least publicly apologize for his exceptional rhino obstinacy ;)
And... Since Frans Bouma has been supporting him exceptionally well there, I think he must accept the same conditions as well.
Obviously, later I'll provide other evidences in addition to this simple game. But I'd like to publish this first.
P.S. Guys, nothing personal. I love reading your blogs and I know you're quite clever persons. But I think if you're standing on your point of view so hard, you are ready to prove it in action. Quite simple.
Kind regards, Alex Yakunin
