PerfChecker Research Project
Contact: andrewust@cse.ust.hk

Researchers

Empirical study

In our project, we conducted a large-scale empirical study to:

understand the common types of performance bugs in real-world Android applications and their impact on user experience (Bug type and impact)
understand how performance bugs manifest in real-world Android applications (bug manifestation)
understand the difficulties of performance debugging and bug fixing as well as what information or tools can ease these tasks (Debugging and bug-fixing effort)
find common patterns of performance bugs in real-world Android applications (common bug patterns)

We release our data and results on this webpage for research purposes.

Note before you proceed: Data that can be accessed from this page are maintained asynchronously with our research paper repository. Should you find inconsistencies or errors in these data, please feel free to contact us. We will check and update the data accordingly (if necessary).

1. Subject information

We selected eight large-scale open-source Android applications for our study. From their bug tracking systems, we identified a total of 70 performance bugs. The subject and bug information are summarized below.

App Name	Category	Size (LOC)	Language	Downloads	# Perf bugs
Firefox	Communication	122.9K	Java, C, C++	10M - 50M	34
Chrome	Communication	77.3K	Java, C++, Python	50M - 100M	19
AnkiDroid	Education	44.8K	Java	500K - 1M	4
K-9 Mail	Communication	76.2K	Java	1M - 5M	3
My Tracks	Health & Fitness	27.1K	Java	10M - 50M	3
c:geo	Entertainment	44.7K	Java	1M - 5M	3
Open GPS Tracker	Travel & Local	18.1K	Java	100K - 500K	2
Zmanim	Books & Reference	5.0K	Java	10K - 50K	2

Notes: 1K = 1,000; 1M = 1,000,000. For Firefox and Chrome, we counted only their code specific to Android.
More information: IDs of our studied 70 performance bugs

We highlight our results below. For more details, please refer to our research paper.

2. Bug types and their impact

We observed three common types of performance bugs in Android applications:

GUI lagging (53/70, 75.7%): These bugs can significantly reduce the responsiveness and smoothness of an application's GUI and prevent user events from being handled in a timely fashion.
Energy leak (10/70, 14.3%): These bugs can cause an application to consume excessive battery power, but the energy consumption brings little benefit to smartphone users.
Memory bloat (8/70, 11.4%): These bugs can cause an application to consume more memory than necessary.

Summary: GUI lagging, energy leak, and memory bloat are three dominant performance bug types in our studied Android applications. Research effort can first be devoted into designing effective techniques to combat them.

2. Bug manifestation

We made several interesting observations related to bug manifestation. These observations demonstrate unique challenges in performance testing and analysis.

Small-scale inputs suffice to manifest performance bugs. We only observed 11 performance bugs (out of 70) that require large-scale inputs to manifest.
Special user interaction needed to manifest performance bugs. More than one third (25/70) of our studied bugs can only manifest after a certain sequence of user interactions happen to the concerned application.
Automated performance oracle needed. Performance bugs rarely cause fail-stop consequences and it is challenging to decide whether an application is suffering from performance bugs automatically. We found three common judgement criteria that have been used in practice: (1) human oracle, (2) product comparison, (3) developers' consencus.
Performance bugs can be platform-dependent. A non-negligible proportion (6/70) performance bugs require specific software or hardware platform to manifest.

Summary: Effective performance testing/analysis requires; (1) new coverage criteria to assess testing adequacy, (2) effective techniques for generating user interaction sequences to manifest performance bugs, and (3) automated oracle to judge performance degradation.

3. Debugging and bug-fixing effort

Summary of findings: Debugging and fixing performance bugs are generally more difficult than debugging and fixing non-performance bugs. Information provided by profilers and performance measurement tools is more helpful for debugging than traditional information like stack trace. Existing profilers expect improvement for automatically analyzing, aggregating, simplifying and visualizing collected runtime profiles.

4. Common bug patterns

We found that performance bugs in Android applications can be very complex and have application-specific root causes. However, we still identified three common bug patterns from 21 of our studied bugs:

Lengthy operations in main thread. Android applications should not block their main thread (used to handle user actions) with heavy tasks. However, when applications become complex, developers tend to leave lengthy operations in main thread unintentionlly. This pattern of bugs can cause GUI laggings.
Wasted computation for invisible GUI. These bugs cause an application to update its GUI when the application itself has been switched to background (that is, the GUI is invisible). This pattern of bugs can cause energy leaks.
Frequently invoked heavy-weight callbacks. Android applications contain many callbacks that will be frequently invoked by the Android OS. These callbacks need to be light-weight. However, many such callbacks have been ill-implemented. They are heavy-weight can significantly slow down concerned applications.

Summary: Our study identified three common performance bug patterns: (1) lengthy operations in main threads, (2) wasted computation for invisible GUI, and (3) frequently-invoked heavy-weight callbacks. Researchers and practitioners should design effective techniques to prevent and detect such performance bugs.

If you have any questions or find any errors in our data or results, please contact Yepang Liu via email andrewust A/T cse D/O/T ust D/O/T hk.