ORMBattle.NET

I just published a set of new articles:

• Rules
• Contributors
• 2 new articles in FAQ.

As some of you know, we've got a set of claims from visitors of this web site. So we decided to compose a list of major of them and describe the decisions we're going to implement that finally must eliminate all the mentioned the issues. So let's go:

1. Tests are wrong - they're meaningless, unrealistic and so on.

We will:

• Write articles & FAQ explaining what we've been testing much deeper, as well as the importance and applicability of results.
• Implement at least one of standard TPC tests (TPC-C / TPC-E).
• Add CUD tests with explicitly disabled batches; change "CUD average" derivative formula.

2. Tests are intentionally written to score DataObjects.Net better; other ORMs are misused on these tests

Ok, let's be fully honest: I think this claim is simply lie. There were no any evidences approving this, and I believe they won't appear. It was initially shown there is nothing to hide for us. You can download the source code of this test suite, DataObjects.NET (it's shipped with full source code as well) and test everything by your own. Run it under debugger\profiler, see what it happens there, find out if something special is done for any of these tests. I'm ready to bet you'll find nothing.

Moreover, as we wrote, this test isn't "designed" for DataObjects.Net architecture - on contrary, it's one of the worst tests for it. E.g. we knew we must loose in materialization test to ADO.NET Entity Framework. I'll explain what sort of framework might be high-scored on our test suite shortly in FAQ.

Anyway, DataObjects.Net is already removed from test results at all - at least to stop any argues around this.

We will:

• Provide links to test files in each participant's page. This will allow all of our visitors to validate the test code.
• Publish full source code on Google Code (btw, already done) and invite test co-authors and experts there.

3. This site is nothing more than DataObjects.NET marketing resource

There is no more DataObjects.Net here. Hopefully, this fixes the main issue.

We want it to be a marketing resource for any good ORM tool on .NET.

• All you need is higher score.
• As I'll shortly explain, no one can get it with just some simple tweaks. The company must be an expert in .NET development & ORM to be well scored here.

4. This site is private and closed resource. Thus there is no room for open discussions here.

Already done:

• Comments are enabled for all the pages.
• Discussion forum is fully open now (earlier it was necessary to register here to access it).
• Test suite source code is shared at Google Code.
• Online source code browser is added to this website.

5. The name of this web site implies agression. Thus it looks like #2 and #3 is true.

Initially we took orm-comparison.net and orm-comparison.org as well. Actually I wanted to give this site a bit unusual name associated with orm comparison to make it easy to remember. Anyway, we'll change its name after some marketing research and visitors polling.

Hopefully, this will help to make ORMBattle.NET better.

P.S. If I forgot about something important, please leave us a comment.

Kind regards,
Alex Yakunin.

In general, we're very glad seeing what's happening. ORMBattle.NET, being just two days newborn, made an effect of exploding bomb - actually, we didn't expect this. We've gathered the most discussable blog posts about it here. Right now I can list just few decisions we've already made and done:

1. Remove DataObjects.Net from results published on this web site.

We decided to avoid arguing with some people believing the benchmarks provided here are designed for this product or intentionally made to market it. Unfortunately, that was the simplest way to prove the idea behind this website and its results are of greater importance for us. There are still tests for it in source code, but we don't publish its results here any more.

2. Remove one more ORM tool previously listed here

No comments.

3. We've implemented all proposed NHibernate test improvements.

If you're interested, new test code is here; most of implemented fixes are mentioned here.

4. Scorecard is updated in accordance with above statements

Check it out. Press F5, if puctures displayed there are the same as before.

5. Project code is shared at Google Code

If you'd like to access the repository, just write to us.

I'll publish all the other news tomorrow.

ORMBattle.NET site is officially live now. If you found useful the information it provides, please don't forget to share the link to us :)

And, as it was promised, articles related to test result analysis will appear here soon.