Firebase Database — how to survive billing issues
Incorrect numbers on billing? Charged too much? No breaks on expenses? Or maybe mysterious usage with an empty database?
Edit: Firebase response with explanation and ‘solution’ at the end.
WELCOME TO FIREBASE MYSTERIOUS BILLING
Where 0 bytes is 1850 bytes (or extra 130 MB daily with an empty database)
You’ve probably you heard about the HomeAutomation article with over 7,000% billing increase, well we experienced numerous issues and we decided to do something about it for others!
I first realized the issues with Firebase Database billing in August 2016— when we built Yawn.wtf — we saw a difference between numbers in Google Cloud Platform billing and in the Firebase console. That wasn’t big issue — we contacted support, after three emails it was clear that numbers in Firebase console were wrong and they were sorry. Done.
Success? Better prepare your wallet.
In December 2016 I made Firebase Database my only data source for my pet-project, Kanarek — a free Android app to monitor air pollution in Poland. In January the app jumped from 150 users to over 100,000 (with over 50k daily active). All that because the topic got mass media attention and due to a usage spike all competitors apps stopped working as they use data directly from the source (governments servers — which stopped working) and I put Firebase in the middle to cache the data. Let me just tell you I was very, very pleased with the growth and usage of the app (avg. rating is 4.6) and how smooth it was for the users.
However the issue is — costs are linearly scalable with usage…
I was sure that the app was using too much data per user, and there was some bug in my code. Also I quickly started to experiment with caching (Firebase SDK built in and Realm) to see how I can optimize it.
You upload you free app, it becomes next “Flappy Bird” and unless you pull the plug out and delete Firebase project, there is no stopping for the expenses. For me, this was really terrifying.
So my second issue with Firebase billing — there is no way to control expenses or have any details on those. In that respect AWS DynamoDB is superior — you set a threshold of transfer per second that you are willing to pay for (also you can use Lambda to auto scale it), and if you are above the limit, your users have to wait.
We will bill you, try to stop us
Third — mysterious 0.4 GB daily usage with no data in the Database. This is quite new and pushed me to write this story.
After having usage without any way to control it and paying a lot for it (1GB = $1) — I moved my database to AWS DynamoDB, which turned out to be a great decision. With greater db speed I got almost 20x cheaper pricing, and no headache to be constantly worried about. But there are few minor trade-offs (probably for next story).
After almost four months since introducing the version with DynamoDB. I decided that remaining users with old version should buckle up and update app. I though that I was still paying $0.4 daily due to those stubborn users.
Well… you should see my face when I removed all data from Firebase Database and was still paying almost the same amount (on the 25th of May I created backups and cleared all DB data).
On the 29th of May, I contacted Firebase support to ask why I’m still paying for non-existing Database usage.
I immediately got response but no explanation, it is 12 days later and still no solution. 29th of May:
Thanks for reaching out. Please allow me to have some time to look into this. I’ll get back to you once I have an update or if I would need additional details from you.
7th of June:
I’m now communicating this to our engineers for additional checking and analysis. I will reach out to you again as soon as there are updates regarding this.
On the 30th of May I updated read/write rules to block any traffic.
The number of connections dropped but usage still remain above 130 MB daily — with an empty DB and all traffic blocked. Huh?
I only wish that it would be possible to stop it somehow without deleting the whole project, and I also hope that I wasn’t paying for 130 MB daily of the artificial transfer ($24) for past six months or that stats weren’t ‘tuned up’ by X% — as it is hard to trust them any longer.
I tried Firebase Profiler — and failed miserably, in 10 minutes time DB got 510 requests (all were denied) and each was… 0 byte size read. So there should be around 70k daily requests, so therefore billed 130 MB daily should result in average request being around 1857 bytes (instead of 0).
Side note — by knowing Firebase project id (ex. from URLs in app requests) you can run server to read any data in the loop, be denied access and bill owner up to $69 monthly — 50 ms for read operation gives around 1.2 m request daily, which (if as in my case 70 k=130 MB) gives 69 GB and $69 bill (to run server= $5), pretty pricey.
Here to help
During that time I realized how much optimization you can do in your app to make it fast for the users, reliable to network issues and optimize your costs. I started that article by saying “we did something about it” and now we are ready to help you with Icelytics.io — we monitor your app data usage and pinpoint exact issues in that particular app versions to tell you how you can optimize your app. As I was whining all the time about the lack of context (which app version? read/writes? what issues with reaching API? Which code method is responsible?) we made sure that all data we present is relevant and as context-oriented as it can only be. Also you can use Icelytics.io with Firebase, AWS, Azure or any SDK or API!
There are number of things that you can do to lower your Firebase Database usage- good data structure is crucial and keeping your data as light as possible in each node is first thing to do. However if you put out a new version of the app with better DB structure — without Icelytics.io you still how no measurable statistics to see how effective it is. We would be glad to help you make your app better, as well more cost-efficient!
Alternatives
There are some, but none without some downsides. Lately introduced Firebase Performance tool — doesn’t allow monitoring Firebase itself, only external APIs are taken into account, and comparing to Icelytics it also lacks context that can be used to find source of the issue and fix it.
Firebase Profiling tool — manually check read/writes on DB side. This is same as getting those logs from support. I’m happy it is here (finally) but still you have no idea which version of the app is responsible for that usage or which screens/code methods, and it is done manually — so if you use it you probably already have serious issues and no data before that. Not to mention that in my issue it didn’t help (and it should do exactly that).
Making your own — this is the way we went and trust us, supporting 10–50k users in one country is very different to run things fast globally. Also we are all about improving our solution — lately we added monitoring of request flow to track time and source of each request (and not only its response).
Firebase explanation
I got on 8th of June
(…) It seems that all of your connections are setup using websockets and the client running your application doesn’t support TLS session resumption. This means for that for each of the connections, it must undergo the full connection establishment which includes downloading the *.firebaseio.com SSL certificate, negotiating SSL handshaking, and finally establish a connection to your Firebase. Once that connection is established, there is also protocol overhead that lets the client know that there is no data. In order to completely eliminate outgoing bandwidth, connection establishments must be removed to prevent any protocol and SSL overhead (…)
And 25$, however those are issues that shouldn’t be there when you use official SDK. As for solution for further bills for clients that didn’t update app since February…
You can downgrade to ensure no more charges are made on the account.
