Why is it almost impossible to have benchmarking tool for smartphone cameras?

As smartphone companies become more competitive with metrical analysation, companies like Geekbench or Antutu give users an in-depth analysis of the system. This analysis is more about the phone’s performance and less on other important functions like camera. But can cameras be analysed accordingly?

What are benchmarking tools?

Antutu Benchmark Score as of February 2020

Benchmarking tools are dedicated softwares that separates single-core and multi-core performance, and workloads simulating to real-world scenarios based on their performing metrics.

Mobile benchmark softwares are application that simply forces the hardware of the smartphone to its limits by putting CPU intensive tasks. This allows the software to check the productivity of the device for the following hardwares:

  • CPU (Central Processing Unit)
  • GPU (Graphics Processing Unit)
  • Memory (RAM – Random Access Memory)
  • UX (User Experience) based on speed

These hardwares are graded, the higher the score the better that device will perform. But yes the performance may still vary due to change in climate, load in CPU & other variable factors.

Now let’s talk about camera benchmark

Similar to processors or actually more than processors people are nowadays impressed by mobile cameras, but how reliable are mobile cameras with all the bulky numbered megapixels they showcase? A 100 or 64 megapixel camera might still not be as good as the iPhone 11 Pro Max’s 12 megapixel.

Testing of a device under extreme lab condition

Unlike processors cameras don’t have such system where it can be pushed to an intensive task, cameras are solely an additional component like speakers but is very different in usage. There are already companies like DxOMark or DPReivew who review cameras based on their usage at different settings. These reviews are not a software generated benchmark but an extensive use of camera by trained professionals in that particular interest. DxOMark has a dedicated indoor laboratory where they conduct maximum of their photography and also choose outdoor sites for advanced photography. This would possibly make them the most scientifically accurate image analysing technology. But the problem is not there.

Are they accurate?

Well no, Smartphones are evolving on a very fast pace primarily focusing on its camera abilities. For eg the Night mode feature, a Huawei Mate 30 Pro can process a better night photography compared to Samsung Galaxy S10 or OnePlus 7 Pro, but to click this perfect shot, you should place the camera for a duration of 30 seconds compared to the 5-10 seconds on S10 or 7Pro. What would be the parameter to rank this feature?

No smartphone camera is perfect in all its features, unlike the processing units. A processing unit is either fast or comparatively slower. Snapdragon 865 is definitely faster than Snapdragon 855+ in every feature, but a camera can vary based on their software optimisation. Some cameras may click a better portrait mode, a better wide-angle mode, a better night-mode, a better astro-photography, a better video, a better vivid in daylight or better in low light, now how can a benchmark score value which of the above is better or even rank accordingly? Lets take an wild example

ParameterPhone APhone B
Portrait Mode7060
Wide-angle8050
Night-mode4090
Astro-photography5090
Video6060
Daylight6050
Low light5070
Total410470

The above assumed benchmark score shows that Phone B can click a better low-light photography while Phone A is good for other performance. Most people prefer portrait mode or wide-angle mode over low light photography, but the benchmark score of Phone B is substantially higher than that of Phone A. On the other hand if the Night-mode’s benchmark scoring is ignored or undervalued, then a biased opinion will be fabricated.

On top of that, the DxOMark also provides smartphone companies with an expensive hardware, software and training test-kit, this however enables company to match the DxOMark parameter and dispense devices accordingly. This makes the camera a mere examination result with no importance given to quality but benchmark score.

Can this problem be solved?

Perhaps, perhaps not. A single benchmarking software can never, or at least in the near future analyse camera scores. This should be done by trained professionals like DxOMark but should not come with an average score for smartphones rather categorise the different abilities of the camera. For eg: the best Portrait mode camera, best Ultra Primary Camera etc. and rank likewise. With which, it would be totally on the user to decide which camera would suffice his needs.

Amal Varghese

A lecturer by profession and an admirer of transitions. I love the way the technological world revolves around me from the introduction of an earpiece to the computing device in a space shuttle, every tiny detail amuses me. I try to put my fascination to words for all the fanatics of this evolution who are ironically hooked to its benefits anyway. Proud to be a part of this generation who saw a tape-recording cassette with 5 songs and a cloud streaming service with a zillion songs inside your pocket.

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this: