Sunsetting MethodsCount.com

So Long, and Thanks for All the Fish!

This post was written collaboratively by me and my dear friend Nicola, who I thank again for having helped me creating this tool.

For those of you that don’t know us or have never used it, MethodsCount is (was?) a service that allows to compute the number of methods of a given library for Android. Each library comes with the methods count, the byte size, and the list of dependencies, each with their own details.

The idea was thus born: a service that was able to do such computations automatically, with a cache that provided instantaneous results for already computed libraries. Perhaps an integration with Android Studio, that would’ve been even more convenient.

The service has been created using Sinatra + Javascript, supported by a MySQL database. We will break it down and observe each component.

At the core of the service we have the methods count. In order to compute this information, we use Gradle to orchestrate the different steps of the computation pipeline:

What’s important to know is that the computation for each library is quite CPU intensive, in particular if the dependencies have themselves not been computed yet: it’s a recursive procedure! This is also why we try to cache as much as possible (more details soon).

The result is a 2 parts backend:

We chose Amazon Web Services to host our backend, mainly because Nicola was very experienced with it. The services are running on different EC2 instances, and have different scaling policies.

In order to simplify the operational effort to give shape to our infrastructure and maintain it, we have leveraged Elastic Beanstalk, keeping the database manually configured on RDS in order to be able to share it between the two services.

The frontend-server communicates with background workers via SQS (the AWS message queuing service).
The background-workers listen to the queue and process the items (i.e., the libraries) as they come, scaling out into multiple parallel workers in case of demand (the Elastic Beanstalk worker environment helped a lot in setting up the autoscaling policies). The frontend component is also autoscaled and multiple instances a load-balanced via an ELB.

This architecture allows us keep the interface responsive while the user is waiting for the result to be computed. The waiting is implemented via polling; a push-based implementation was in the works, but never saw light of day. The frontend-server, once the library computation request has been forwarded to the workers, polls the database for the result.
This approach has saved us already once when we experienced a partial outage: a single background worker went out of memory (the library graph computation is also memory intensive), but it wasn't replaced by the autoscaling group due to a badly configured health check. The website kept working until we noticed the problem and, once the workers were back up, all the pending library requests in the queue have been processed. No data loss, user only partially affected (already computed library could still be fetched).

We use a bash script leveraging the AWS CLI in order to update the SDK we use in the workers. Given that the update takes quite a few resources and time, it would not be ideal to execute it every time a new instance is created. We therefore bake the updated SDK into an Amazon Machine Image and essentially update the Elastic Beanstalk configuration for the EC2 instance.

We just recently dropped the support for the Android Studio integration, due to lack of time for maintenance and excess of consumed resources.

This is a brief summary of what MethodsCount has been able to achieve.

Total number of libraries (unique by FQN): 55,364
Total number of unique libraries (version is discarded): 16,667
Total number of computed methods: 71,968,176

Total pageviews: 6,044,143
Total sessions: 573,596
Total users: 143,702
Average session duration: 00:05:16

We have also calculated some tables that might be interesting for you to take a look at:

There are a few reasons why we decided to end support for MethodsCount.

Keeping in mind the architecture that we described before and the CPU intensive operations that we needed to run to compute the number of methods, MethodsCount was not cheap to maintain.

We tried a donation-based sustainability model, but it clearly didn’t work (but thanks for those of you who did it anyway!).

We also tried to seek sponsorships from Google and Amazon, but without much success.

Two years ago, when the website was launched, managing to stay below the 65K methods limit was a big deal and required a big effort. Now, as we move towards a brighter future, this need is no longer a major priority for the Android community.

This is mainly a consequence of the previous point. Given that the motivation was beginning to fade, so was our time committed to this project.

We decided to open-source the code for MethodsCount, both for frontend and backend, as well as the database that contains all the statistics that we have collected so far. Feel free to take a look around!

The website is now in read-only mode, meaning that no new libraries can be computed. We have no immediate plans for a total shut down, but that might change in the future.

We would like to thank you all for showing appreciation to the service and to us! It was really heartwarming to see that something we created was used by more than 2000 people on a daily basis.

Even more so, a huge thanks to all of you who donated to keep the service running. No matter how small the donation was, it meant a lot to us!

But in all the sadness, a silver lining should brighten our day:

Related posts:

The deep melodious tone of the narrator in Theodore’s dreams said. This type of lucent dreaming only happened during these episodic sleep things. Theodore asked his mom once when he was younger, if…

Series of five Rwandan government tweets denying any agreement to accept expelled African refugees from Israel The government of Rwanda’s official Twitter account pumped out five tweets miraculous…

The first thing you should consider when looking for a personal injury lawyer is their experience. A lawyer who has handled cases similar to yours in the past will have the knowledge and skills…