Although we did not collect the test data and so cannot be sure that all necessary variables were controlled across tests, we did use our standard methodology for analysis within the GameBench dashboard.
This involves, for example, using the Chart Selection feature to highlight gameplay sections only, ignoring menus and load screens. It also means we only recorded minimum frame rates that occured while gameplay animation was onscreen (including animation that took place behind an in-game prompt or pop-up).
GameBench Labs is the service side of GameBench Ltd, offering:
- Data collection, using crowd data from users of the free version of GameBench, plus hired testers in a range of markets who have been trained to use GameBench tools in combination with a strict game-testing methodology;
- Product ratings, based on GameBench’s industry-recognised system for objectively grading gaming experiences, useful for a top-level summary of results, competitor analysis, internal KPI-setting and external marketing;
- Performance analysis, including competitor analysis and highlighting likely bottlenecks and optimisation issues;
A top-tier game studio was keen to know how their latest casual farming game compared to a rival in the same genre. We provided our industry-standard ratings that showed that they did have a serious performance issue on Android (shown in the grid below).
We found consistent graphical under-performance on Android, relative to both the competing title and to our our own expectations based on GameBench crowdsourced data.
Our profiling data shows that this under-performance took the form of low and excessively variable frame rates on all three Android devices tested, such that the user would be likely to perceive jerky animations throughout a gameplay session. While the competing game was also far from perfect, suffering some significant frame rate drops, it was much smoother overall, as highlighted by our ratings.
On the basis of our findings, we were able to point towards a likely CPU optimisation issue as the root cause. The game-under-test was a casual game, but was as heavy on the CPU as a high-intensity game like PUBG Mobile, using around 3x more CPU than we’d expect. In turns, this appeared to cause the GPU to be under-utilised, versus what we saw with the rival game also versus our observations from other casual games like Monument Valley 2 or Angry Birds.
Our report helped the QA team to get the necessary attention and resources to dig deeper and begin to solve the issue.