There is a set of questions like this we got. Here are some of them:
- "Why you don't use executable DML queries in insert / update / delete tests"
- "Why you use fetch in a loop? This is well-known bad practice, or just invalid scenario. You must use query."
- "Why queries in query test fetch just one instance? Normally query fetches many of them."
The answers:
1. I don't consider the tests are unrealistic. Let's imagine we test CPUs, and want to know the cost of basic operations there (anyone knows these costs for CPUs!) - like add, subtract, store, load and so on. To do this, we normally repeat the same operation in a loop to determine the cost, because measuring just one of such operation won't be precise enough.
That's why we repeatedly execute the operations for many times. Yes, in real life you don't do this - these operations are normally running unordered. On the other hand, most part of real-life ORM-based application operation time is spent on:
- Queries & materialization. If 80/20 rule is applicable here, ~ 80% of time application waits for the answer from ORM is spent here.
- CUD operations. Here are another 20%, although in many particular cases (data load & import) it reaches up to 100%
Have I forgot sometihng? Will it get additional 20% among the above items? If no, our test tests are very realistic, because they measure exactly these operations.
2. I dislike the idea to use a special API that will suit for our tests. To illustrate, why, let's return back to CPU tests:
- We agreed to test add operation by the following test: we add constant number to accumulator and measure the time spent on N=1K of such operations.
- But as you see in quotes above, people insist we must use multiply operation to measure the performance of add operation, because in this case our test will complete faster on all CPUs, that support multiply.
Seems absolutely strange, yes? What is the result of such measurement? Yes, 1/Nth of cost of multiply operation. Is this result related to add operation? No.
More important question: why they propose to use multiply operation? Because in this case CPU they push forward will be faster on add test (although actually it isn't) even if the the result won't reflect add operation time at all. One more reason of such proposal is possibility to replace add to multiply in this test: both N additions of constant and 1 multiply operation lead to the same result! Obviously, it's just a particular case when multiply can replace add. But these people behave like they don't see this.
Ok, we can rewrite our CPU test to add completely different numbers there. No more multiply, and the same results. We don't do this just because it's rediculous! We must fix a test for some general API in the way they it shouldn't be possible to adopt it to some special API? That's simply crazy. Very next day some CPU vendor will invent a special operation for the whole test and will say you must use it - by the same reason. Or better, "Just because I want to win!"
Adopting this to our test suite, they propose to use executable DML to measure update & delete time. What they're going to measure by this way? They are going to measure the performance of SQL Server executing these operations, but not ORM performance. So such test won't be related to ORM at all.
That's why I think such adoptions for a particular ORM API must not appear here.
3. It's intentional to minimize load on database server on our tests.
We measure the efficiency of intermediate layer, thus we must make all the layers behind it operating as fast as possible - to measure mainly the operation time of that intermediate layer.
That's why, for example, query test is running a simple query loading a single instance.
