For people finding this thread via web search in the future:
screen.studio is macOS screen recording software that checks for updates every five minutes. Somehow, that alone is NOT the bug described in this post. The /other/ bug described in this blog is: their software also downloaded a 250MB update file every five minutes.
The software developers there consider all of this normal except the actual download, which cost them $8000 in bandwidth fees.
To re-cap:
Screen recording software.
Checks for updates every five (5) minutes. That's 12 times an hour.
I choose software based on how much I trust the judgement of the developers. Please consider if this feels like reasonable judgement to you.
vrosas 7 minutes ago [-]
When I built an app that “phones home” regularly, I added the ability for the backend to respond to the client with an override backoff that the client would respect over the default.
VWWHFSfQ 8 minutes ago [-]
I would be so embarrassed about this bug that I would be terrified to write it up like this. Also admitting that your users were forced to download 10s or 100s of gigabytes of bogus updates nearly continuously. This is the kind of thing that a lot of people would just quietly fix. So kudos (I guess) to blogging about it.
abstractspoon 1 hours ago [-]
I find it ludicrous that the developers of an app as insignificant as a screen recorder would think it's necessary to check for updates every 5 minutes.
Once a day would surely be sufficient.
EvanAnderson 26 minutes ago [-]
> Once a day would surely be sufficient.
Weekly or monthly would be sufficient. I'd also like "able to be disabled manually, permanently" as an option, too.
dist-epoch 21 minutes ago [-]
You can use that as a hidden way of tracking how many active users you have at any time.
Good way of showing adoption and growth.
mystified5016 1 minutes ago [-]
You can still do that with daily, weekly, or monthly checks.
Nobody under any circumstances needs usage stats with 5 minute resolution. And certainly not a screen recorder.
panki27 5 minutes ago [-]
That's stretching my definition of "good" quite a bit.
Yeri 47 minutes ago [-]
a thousand times this.
amelius 25 minutes ago [-]
one time should be sufficient
jmull 1 hours ago [-]
I'm pretty conservative about adopting third-party libraries (due to the long-term issues each one has the potential to cause), but an app updater is probably worth it.
It's just tricky, basically one fat edge case, and a critical part of your recovery plan in case of serious bugs in your app.
(This bug isn't the only problem with their home-grown updater. Checking every 5 min is just insane. Kinda tells me they aren't thinking much about it.)
Zambyte 27 minutes ago [-]
Or better yet, let the system package manager do it's job.
wqaatwt 18 minutes ago [-]
You’d be forced to use Apple’s App-Store, though? I don’t think there is an other package manager
Zambyte 10 minutes ago [-]
As far as system package managers go, yeah. That's part of the price you pay for choosing Apple (Knows Best) TM. There is brew, nix and the like for applications on MacOS too though.
madeofpalk 6 minutes ago [-]
Apple doesn't "know best" - it's just that that is what the system package manager is.
You can use whatever you want outside of the App Store - most will use Sparkle to handle updates https://sparkle-project.org/. I presume Windows is similar.
jve 8 hours ago [-]
> Screen Studio is a screen recorder for macOS. It is desktop app. It means we need some auto-updater to allow users to install the latest app version easily.
No, it doesn't mean that.
Auto updater introduced series of bad outcomes.
- Downloading update without consent, causing traffic for client.
- Not only that, the download keeps repeating itself every 5 minutes? You did at least detect whether user is on metered connection, right... ?
- A bug where update popup interrupts flow
- A popup is a bad thing on itself you do to your users. I think it is OK when closing the app and let the rest be done in background.
- Some people actually pay attention to outgoing connections apps make and even a simple update check every 5 minutes is excessive. Why even do it while app is running? Do on startup and ask on close. Again some complexity: Assume you're not on network, do it in background and don't bother retrying much.
- Additional complexity for app that caused all of the above. And it came with a price tag to developer.
Wouldn't app store be perfect way to handle updates in this case to offload the complexity there?
HelloNurse 7 hours ago [-]
App store updates are perfect: no unnecessary complications, no unnecessary work (assuming Screen Studio is published and properly updated in the app store), and the worst case scenario is notifications about a new Screen Studio version ending up in a Screen Studio recording in progress.
Thinking of it, the discussed do-it-yourself update checking is so stupid that malice and/or other serious bugs should be assumed.
amelius 17 minutes ago [-]
As a user I hate auto updates. It feels like someone pulling the rug from under me.
Nition 5 hours ago [-]
While we're listing complaints... 250MB for a screen recorder update?
yojo 18 minutes ago [-]
That’s pretty much the floor for an Electron app.
If you’re a small shop or solo dev, it is real hard to justify going native on three platforms when electron gives it for (near) free. And outside of HN, no one seems to blink at a 250MB bundle.
There are alternatives like Tauri that use the system browser and allow substantially smaller bundles, but they’re not nearly as mature as Electron, and you will get cross platform UI bugs (some of which vary by user’s OS version!) from the lack of standardization.
crest 25 minutes ago [-]
And even when nothing changed?!? Fucking lazy developers aka "I have an idle ≥1Gb/s pipe to the download server". What happened to rsync/zsync/zstd (with dictionary)? There are so many good tools freely available to reduce wasted bandwidth when you insist on reinventing the wheel. sigh
dist-epoch 19 minutes ago [-]
> Wouldn't app store be perfect way to handle updates
But then the HN crowd would complain "why use an app store? that's gate keeping, apple could remove your app any day, just give me a download link, and so on..."
You literally can't win.
wqaatwt 15 minutes ago [-]
You can? Don’t check for updates every 5 minutes. Daily or even weekly would be sufficient for an app like this (if auto-updater is even necessary at all.. just show a notification)
socalgal2 8 hours ago [-]
Does the app store handle staged rollouts?
That was a thing I thought was missing from this writeup. Ideally you only roll up the update to a small percent of users. You then check to see if anything broke (no idea how long to wait, 1 day?). Then you increase the percent a little more (say, 1% to 5%) and wait a day again and check. Finally you update everyone (who has updates on)
dahcryn 5 hours ago [-]
yes obviously something as mature as the App store supports phased rollout. I believe it is even the default setting once you reach certain audience sizes. Updates are always spread over 7 days slowly increasing the numbers
djxfade 6 hours ago [-]
Yes it does support this
albert_e 1 hours ago [-]
What about the bandwidth burned needlessly for thousands of users on their data plans.
At some scale such careless mistakes are going to create real effects for all users of internet through congestion as well.
If this was not a $8000 mistake but was somehow covered by a free tier or other plan from Google Cloud, would they still have considered it a serious bug and fixed it as promptly?
How many such poor designs are out there generating traffic and draining common resources.
bee_rider 7 minutes ago [-]
They mention specifically handling the situation for one user. So, I guess it is a case-by-case thing.
vachina 2 minutes ago [-]
Does the developer release a tag for every ctrl+s.
spaqin 8 hours ago [-]
I would also put into question if you _really_ need to check for updates every 5 minutes. Once per startup is already enough, and if you're concerned about users who leave it on for days, it could easily be daily or even less often.
stevage 1 hours ago [-]
It's absolutely way too frequent.
Their users do not care about their screen recording studio anywhere near as much as the devs who wrote it do.
Once a month is probably plenty.
Personally, I disable auto-update on everything wherever possible, because the likelihood of annoying changes is much greater than welcome changes for almost all software I use, in my experience.
Lammy 8 hours ago [-]
A 5 minute update check interval is usage-reporting in disguise. Way fewer people would turn off a setting labeled “check for updates” than one labeled “report usage statistics”.
bilekas 8 hours ago [-]
Don’t give them ideas!!
blitzar 1 hours ago [-]
never attribute to malice what can be attributed to incompetence
Lammy 6 minutes ago [-]
No. Eradicate this line of thinking from your brain. If the outcome is the same then the intent doesn't matter.
YetAnotherNick 8 hours ago [-]
Do they say that they don't do any usage reporting?
TowerTall 6 hours ago [-]
from their FAQ on the buttom of the fronpage:
Screen Studio can collect basic usage data to help us improve the app, but you can opt out of it during the first launch. You can also opt out at any time in the app settings.
Spivak 8 hours ago [-]
Eh, this one is probably ignorance over malice. It's super common to see people who need to make an arbitrary interval choice go with 300 out of habit.
sixtyj 1 hours ago [-]
Check for updates every 5 minutes is a bug itself ;)
It is sort of fun (for $8,000) as it was “just” a screenshotter, but imagine this with bank app or any other heavily installed app.
All cloud providers should have alerts for excessive use of network by default. And they should ask developers if they really want to turn alerts off.
I remember Mapbox app that cost much more, just because provider did charge by months… and it was a great dispute who’s fault it was…
karhuton 8 hours ago [-]
To be as user friendly as possible, always ask if user wants automatic background updates or not. If you can’t update without user noticing it, please implement manual updates as two mechanisms:
1) Emergency update for remote exploit fixes only
2) Regular updates
The emergency update can show a popup, but only once. It should explain the security risk. But allow user to decline, as you should never interrupt work in progress. After decline leave an always visible small warning banner in the app until approved.
The regular update should never popup, only show a very mild update reminder that is NOT always visible, instead behind a menu that is frequently used. Do not show notification badges, they frustrate people with inbox type 0 condition.
This is the most user friendly way of suggesting manual updates.
You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
zveyaeyv3sfye 5 hours ago [-]
> You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
That's not an user issue tho, it's a "packaging and distribution of updates" issue which coincidentally has been solved for other OS:es using a package manager.
wqaatwt 9 minutes ago [-]
Or a developer problem when they keep updating their apps every few days for no apparent reason..
adrianN 2 hours ago [-]
Getting used to changes is not something a package manager can help with.
trollbridge 2 hours ago [-]
And if it is necessary, the proper way to do this is via DNS with a record with a TTL less than 5 minutes, not pinging some webserver.
This could have easily been avoided by prompting the user for an update, not silently downloading it in the background... over and over.
tom1337 6 hours ago [-]
I'd also question if the updater needs to download the update before the user saying they want it. Why not check against a simple endpoint if a newer version is available and if so, prompt the user that an update could be downloaded and then download it. This would also allow the user to delay the update if they are on metered connections.
If the update interval had been 1 day+, they probably wouldn't have noticed after one month when they had a 5 minute update check interval.
m3adow 8 hours ago [-]
First thing I thought as well. Every 5 minutes for a screen recording software is an absurd frequency. I doubt they release multiple new versions per day.
ghurtado 8 hours ago [-]
IIRC, Every 5 minutes used to be the standard interval between email checks, back in the days of dialup and desktop email clients.
How the times have changed ..
lucb1e 8 hours ago [-]
It's near-instant now not usually because of more incessant polling, but because it simply keeps the connection open (can last many hours without sending a single byte, depending also on the platform) and writes data onto it as needed (IMAP IDLE). This has gotten more efficient if anything
pjmlp 8 hours ago [-]
And because how expensive they were in Portugal, I never done it, it was always on manual.
wodenokoto 7 hours ago [-]
Depends on the application. I have my browser running for months at a time.
atoav 8 hours ago [-]
Yeah but that should be a variable anyways. Maybe even a variable provided by the server. But in this case it should be on demand. with the old version cached and only downloading the new one when there is a new version once a day.
atoav 8 hours ago [-]
Yeah but that should be a variable anyways. Maybe even a variable provided by the server.
ValdikSS 27 minutes ago [-]
I'm running an anti-censorship proxy service which uses Proxy Auto-Configuration (PAC) file which you can configure OS-wide or in the browser.
If the file contains invalid JS (syntax error, or too new features for IE on Win7/8), or if it's >1MB (Chromium-based browsers & Electron limit), and the file is configured system-wide, then EVERY APP which uses wininet starts flooding the server with the requests over and over almost in an endless loop (missing/short error caching).
Over the years, this resulted in DDoSing my own server and blackholing its IP on BGP level (happened 10+ times), and after switching to public IPFS gateways to serve the files, Pinata IPFS gateway has blocked entire country, on IPFS.io gateway the files were in top #2 requests for weeks (impacting operational budget of the gateway).
All of the above happens with tight per-IP per-minute request limits and other measures to conserve the bandwidth. It's used by 500 000+ users daily. My web server is a $20/mo VPS with unmetered traffic, and thanks to this, I was never in the situation as the OP :)
donatj 8 hours ago [-]
I am always kind of a stickler about code reviews. I once had a manager tell me that I should leave more to QA with an offhand comment along the lines of "what is the worst that could happen" to which I replied without missing a beat "We all lose our jobs. We are always one bad line of code away from losing our jobs"
The number of times I have caught junior or even experienced devs writing potential PII leaks is absolutely wild. It's just crazy easy in most systems to open yourself up to potential legal issues.
canucker2016 3 hours ago [-]
...And if there's no one around to review the code?
The website makes it seem like it's a one person shop.
alias_neo 33 minutes ago [-]
When I work on my own code, at home, with no-one to assist or review, I write tests, and open a PR anyway, and review it myself, sometimes the next day with fresh eyes, or even 10 minutes later after a quick walk in and out of the room and a glass of water.
If you're not confident you can review a piece of code you wrote and spot a potentially disastrous bug like the one in OP, write more tests.
jarym 8 hours ago [-]
Just amazed that ‘better testing’ isn’t one of the takeaways in the summary.
Just amazed. Yea ‘write code carefully’ as if suggesting that’ll fix it is a rookie mistake.
So so frustrating when developers treat user machines like their test bed!
fifilura 8 hours ago [-]
Contrarian approach: $8000 is not a lot in this context. What did the CEO think of this? Most of the time it is just a very small speed bump in the overall finances of the company.
Avoidable, unfortunate, but the cost of slowing down development progress e.g. 10% is much higher.
But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
stevage 1 hours ago [-]
I'm sure it cost a lot more than $8000. That was only the direct visible cost to them. There were likely users hit with costs for the additional downloads, who never even knew what was the issue. Users working on a mobile hotspot who had to pay for extra data etc etc.
hbsbsbsndk 2 hours ago [-]
I worked on a product where there was basically no automated testing, just a huge product surface to click around with a bunch of options. Because of technical debt some of the options would trigger different code paths, but it was up to the developer to memorize all the code paths and test accordingly.
After I shipped a bug the Director of Engineering told me I should "test better" (by clicking around the app). This was about 1 step away from "just don't write bugs" IMO.
stevage 1 hours ago [-]
Yep, my first job was at a company like that. Huge Windows desktop app built in Delphi. No automated testing of any kind. No testing scripts either. Just a lot of clicking around.
cryptonym 3 minutes ago [-]
My first job was exactly that, selling windows app in Delphi. I joined the new team working on .net windows apps and we had an army of people clicking on UI all day long.
They maintained their "test plan" on a custom software where they could report failures.
TBH, that was well done for what it was but really called for automation and lacked unit-testing.
Klaster_1 7 hours ago [-]
How do you adjust your testing approach to catch cases like this? In my experience, timing related issues are hard to catch and can linger for years unnoticed.
doix 7 hours ago [-]
I would mock/hook/monkey patch/whatever the functions to get the current time/elapsed time, simulate a period of time (a day/week/month/year/whatever), make sure the function to download the file is called the correct amount of times. That would probably catch this bug.
Although, after such a fuck up, I would be tempted to make a pre-release check that tests the compiled binary, not any unit test or whatever. Use LD_PRELOAD to hook the system timing functions(a quick google shows that libfaketime[0] exists, but I've never used it), launch the real program and speed up time to make sure it doesn't try to download more than once.
>Just amazed that ‘better testing’ isn’t one of the takeaways in the summary.
I don't get the impression they did any testing at all.
indymike 2 hours ago [-]
We just put a header for version in our app, and when we deploy new code the client checks against the version header and upgrades if the version is mismatched. No extra get requests required. Bonus: we just use the last git commit hash as the version. Stupid simple.
Hobadee 27 minutes ago [-]
Why is nobody talking about what a shady business practice it is that cloud providers don't alert you to this kind of overage by default? Sure, you can set up alerts, but when you go 10x over your baseline in a short period of time, that should trigger an alert regardless of your configured alerts.
bee_rider 2 minutes ago [-]
They could compare against the baseline, I guess.
In the grand scheme of things, $8k is not much money for a business, right? Like we can be pretty sure nobody at Google said “a-ha, if we don’t notify the users, we will be able sneak $8k out of their wallets at a time.” I think it is more likely that they don’t really care that much about this market, other than generally creating an environment where their products are well known.
firesteelrain 23 minutes ago [-]
Or treat it like the stock market and shut it down.
sevg 8 hours ago [-]
Why in the world does it need to check for updates every 5 minutes?
The author seemed to enjoy calculating the massive bandwidth numbers, but didn’t stop to question whether 5 minutes was a totally ridiculous.
Fokamul 28 minutes ago [-]
Truth to be told, Google Cloud console is horrible mess for new people, who just wants quickly setup API, pay and don't have time to care about it anymore.
Well, you should hire contractor to set console for you.
"Designed for MacOS", aah don't worry, you will have the money from apes back in the no time. :)
danpalmer 8 hours ago [-]
> We decided to take responsibility and offer to cover all the costs related to this situation.
Good on them. Most companies would cap their responsibility at a refund of their own service's fees, which is understandable as you can't really predict costs incurred by those using your service, but this is going above and beyond and it's great to see.
weird-eye-issue 8 hours ago [-]
"Luckily, it was not needed"
stevage 1 hours ago [-]
I'm really surprised this could happen. As they note:
> Write your auto-updater code very carefully.
You have to be soooo careful with this stuff. Especially because your auto-updater code can brick your auto-updater.
It looks like they didn't do any testing of their auto update code at all, otherwise they would have caught it immediately.
moi2388 5 hours ago [-]
Why on earth are you checking for updates every 5 minutes to begin with?!
Seriously this alone makes me question everything about this app.
dkdbejwi383 1 hours ago [-]
Probably product owner wants to show off a nice chart at their next meeting showing how quickyl users upgrade, as some kind of proxy metric for "engagement"
The scale is astounding. I was briefly interested in the person that caused the error then immediately realized it was irrelevant because if a mechanism doesn't exist to catch an issue like that, then any company is living on borrowed time.
voidUpdate 7 hours ago [-]
whyyy does wikipedia not redirect mobile links to the desktop website when you have a desktop UA?
xigoi 6 hours ago [-]
Why do they have a separate mobile website at all instead of writing proper CSS to make one website work on all devices?
wodenokoto 6 hours ago [-]
Because people on desktops asking for the mobile site should be able to view the mobile site.
The url specifically asks Wikipedia to serve the mobile site.
voidUpdate 5 hours ago [-]
Well when I follow a desktop link on my phone, it redirects me to the mobile version, despite the URL specifically asking to serve the desktop site, it just doesn't work the other way around. Plus I never asked to see the mobile site, I followed a link someone else posted
>Add special signals you can change on your server, which the app will understand, such as a forced update that will install without asking the user.
I understand the reasoning, but that makes it feel a bit too close to a C&C server for my liking. If the update server ever gets compromised, I imagine this could increase the damage done drastically.
saretup 8 hours ago [-]
Not to mention the cost users paid to download 250 MB every 5 minutes.
ghurtado 8 hours ago [-]
It seems a bit self centered to make their lost $8000 the focus of the article.
The title should have been: "how a single line of code cost our users probably more than $8000"
stevage 1 hours ago [-]
Totally. I live in a place where many (most?) ISP plans have limited monthly downloads. I'd be so pissed off if my monthly allowance was blown by this series of boneheaded decisions.
pests 8 hours ago [-]
It was mentioned, at the bottom. One customer even had their ISP cancel their service.
zelon88 30 minutes ago [-]
> As a designer, I value the experience product I create provides to the users. And this was not even a bad experience; it was actually harmful.
$229 per year on a closed source product and this is the level of quality you can expect.
You can have all the respect for users in the world, but if you write downright hazardous code then you're only doing them a disservice. What happened to all the metered internet plans you blasted for 3 months? Are you going to make those users whole?
Learning from and owning your mistake is great and all, but you shouldn't be proud or gloating about this in any way, shape, or form. It is a very awkward and disrespectful flex on your customers.
Fokamul 24 minutes ago [-]
Did you see? It's "Designed for macOS"
I would put premium edition for 3999$ at least.
aranw 6 hours ago [-]
I have Screen Studio and I don't leave it open but all I wish for now is that you disable the auto updater. Provide an option for it to be disabled and allow for manual update checking. Checking for an update every 5 minutes is total overkill and downloading the update automatically is just bad. What if I was on mobile internet and had limited bandwidth and usage. The last thing I want is an app downloading it's own update without my consent and knowledge.
kimbernator 42 minutes ago [-]
This cost -you- $8000. It probably cost users a lot more.
huksley 5 hours ago [-]
They had no cost usage alerts. So they even did not know that the thing was happening, just realized with the first bill.
I think that is the essence of what is wrong with the cloud costs. Defaulting to possibility for everyone to scale rapidly while in reality 99% have quite predictable costs month over month.
hardwaresofton 8 hours ago [-]
Bugs are great chances to learn.
What might be fun is figuring out all the ways this bug could have been avoided.
Another way to avoid this problem would have been using a form of “content addressable storage”. For those who are new, this is just a fancy way of saying make sure to store/distribute the hash (ex. Sha256) of what you’re distributing and store it on disk in a way that content can be effectively deduplicated by name.
It’s probably not so easy as to make it a rule, but most of the time, an update download should probably do this
ghurtado 8 hours ago [-]
> out all the ways this bug could have been avoided.
The most obvious one is setting up billing alerts.
Past a certain level of complexity, you're often better off focusing on mitigation that trying to avoid every instance of a certain kind of error.
HelloNurse 3 hours ago [-]
Note that billing alerts protect against unexpected network traffic, not directly against bugs and bad design in the software. Update checking remains a terrible idea.
misiek08 1 hours ago [-]
So it is 2PB less of bytes written lifetimes on users disks? Interesting to count that.
pornel 8 hours ago [-]
It would also be nice if the update archive wasn't 250MB. Sparkle framework supports delta updates, which can cut down the traffic considerably.
mathverse 8 hours ago [-]
This is an electron app.
dahcryn 5 hours ago [-]
which is their design choice, not an obligation.
Electron really messed up a few things in this world
bryanrasmussen 8 hours ago [-]
I read that as "A single line of code costs $8000" which, from the comments seems like a few others had the same thought. Reading the article it is not costs and the original title is "One line of code that did cost $8,000", so as some others have pointed out it is a bug that cost $8000.
stevage 1 hours ago [-]
I was expecting it to be about a good line of code that cost $8,000 in development time to write, which might be an interesting story.
indrex 5 hours ago [-]
Plenty of (valid) criticism in the comments, but I appreciate the developer for publishing it.
stevage 1 hours ago [-]
I feel a bit iffy about turning the shitty experience you imposed on your users into content for your blog.
pandemic_region 13 minutes ago [-]
This 1000 times. It takes courage to open up to mistakes. As a relatively young industry, we have a lot to learn still to move away from the instinctive blaming culture surrounding such failures. In this case, it's only a file being downloaded a couple of times, nobody died or got injured.
For those interested in this topic, and how other industries (e.g. Airline industry) deal with learning from or preventing failure: Sidney Dekker is the authority in this domain. Things like Restorative Just Culture, or Field guide to understanding human error could one day apply to our industry as well: https://sidneydekker.com/books.
watusername 1 hours ago [-]
Needs (2023) in the title.
jumploops 8 hours ago [-]
Oh boy, I know of at least one case where a single line of code cost ~$500k…
Curious where the high-water mark is across all HNers (:
explodes 6 hours ago [-]
Others have reported higher already, but for data:
Our team had a bug that cost us about $120k over a week.
Another bug running on a large system had an unmeasurable cost. (Could $K, could be $M)
agos 6 hours ago [-]
I would be surprised if half of the user on this site did _not_ create or personally see a bug where a line cost way more than $8000
short_sells_poo 7 hours ago [-]
$1.2mln, gone in about 30 minutes.
insin 7 hours ago [-]
Knowing where to put the line: $7999 (is sadly not the story)
poleguy 8 hours ago [-]
Ever consider not using cloud for everything? Hosting this on traditional hosting would have limited the problem and the cost.
M95D 8 hours ago [-]
And in that case, the problem would not be discovered until 1) someone opened a bug report, which rarely happens, because any competent user would just disable auto-updates, and 2) that bug report would be investigated, which also rarely happens.
cess11 6 hours ago [-]
It's not like you are forbidden to monitor your services just because you didn't put them in big clown.
I assume most of that 2PB network traffic was not egress, right? Otherwise how did it "only" cost you $8k on Google Cloud?
Even at a cost of 0.02$ per GB, which is usually a few times lower than the actual prices I could find there, that would still result in an invoice of about $40k...
byyll 7 hours ago [-]
I'll let my employer know to update my salary or reduce my workload.
navigate8310 8 hours ago [-]
So did you pay or Google showed you mercy by chewing their potential earnings?
cyprx 8 hours ago [-]
meanwhile the CTOs plan to apply AI into their production codebases :)
bilekas 8 hours ago [-]
> While refactoring it, I forgot to add the code to stop the 5-minute interval after the new version file was available and downloaded.
I’m sorry but it’s exactly cases like these that should be covered by some kind of test, especially When diving into a refactor. Admittedly it’s nice to hear people share their mistakes and horror stories, I would get some stick for this at work.
jer0me 8 hours ago [-]
(2023)
jiggawatts 8 hours ago [-]
These articles are great, but I have to one-up the blog: I recently helped a small dev team clean up a one-line mistake that cost them $95,000... which they didn't notice for three months.
The relevance is that instead of checking for a change every 5 minutes, the delay wasn't working at all, so the check ran as fast as possible in a tight loop. This was between a server and a blob storage account, so there was no network bottleneck to slow things down either.
It turns out that if you read a few megabytes 1,000 times per second all day, every day, those fractions of a cent per request are going to add up!
nikanj 9 hours ago [-]
With public sector procurement, $8000 is a pretty standard price for a line of code.
ant6n 7 hours ago [-]
Do you mean "a" line of code, or "each" line of code?
nikanj 5 hours ago [-]
A dead-simple 1000-line app? $8 million from Accenture, IBM or similar
silverfrost 7 hours ago [-]
In other news a screen recorder app is a 250MB (presumably compressed) download...
zveyaeyv3sfye 5 hours ago [-]
FWIW, OBS is ~150 MB, not an electron app and actually open source.
screen.studio is macOS screen recording software that checks for updates every five minutes. Somehow, that alone is NOT the bug described in this post. The /other/ bug described in this blog is: their software also downloaded a 250MB update file every five minutes.
The software developers there consider all of this normal except the actual download, which cost them $8000 in bandwidth fees.
To re-cap: Screen recording software. Checks for updates every five (5) minutes. That's 12 times an hour.
I choose software based on how much I trust the judgement of the developers. Please consider if this feels like reasonable judgement to you.
Once a day would surely be sufficient.
Weekly or monthly would be sufficient. I'd also like "able to be disabled manually, permanently" as an option, too.
Good way of showing adoption and growth.
Nobody under any circumstances needs usage stats with 5 minute resolution. And certainly not a screen recorder.
It's just tricky, basically one fat edge case, and a critical part of your recovery plan in case of serious bugs in your app.
(This bug isn't the only problem with their home-grown updater. Checking every 5 min is just insane. Kinda tells me they aren't thinking much about it.)
You can use whatever you want outside of the App Store - most will use Sparkle to handle updates https://sparkle-project.org/. I presume Windows is similar.
No, it doesn't mean that.
Auto updater introduced series of bad outcomes.
- Downloading update without consent, causing traffic for client.
- Not only that, the download keeps repeating itself every 5 minutes? You did at least detect whether user is on metered connection, right... ?
- A bug where update popup interrupts flow
- A popup is a bad thing on itself you do to your users. I think it is OK when closing the app and let the rest be done in background.
- Some people actually pay attention to outgoing connections apps make and even a simple update check every 5 minutes is excessive. Why even do it while app is running? Do on startup and ask on close. Again some complexity: Assume you're not on network, do it in background and don't bother retrying much.
- Additional complexity for app that caused all of the above. And it came with a price tag to developer.
Wouldn't app store be perfect way to handle updates in this case to offload the complexity there?
Thinking of it, the discussed do-it-yourself update checking is so stupid that malice and/or other serious bugs should be assumed.
If you’re a small shop or solo dev, it is real hard to justify going native on three platforms when electron gives it for (near) free. And outside of HN, no one seems to blink at a 250MB bundle.
There are alternatives like Tauri that use the system browser and allow substantially smaller bundles, but they’re not nearly as mature as Electron, and you will get cross platform UI bugs (some of which vary by user’s OS version!) from the lack of standardization.
But then the HN crowd would complain "why use an app store? that's gate keeping, apple could remove your app any day, just give me a download link, and so on..."
You literally can't win.
That was a thing I thought was missing from this writeup. Ideally you only roll up the update to a small percent of users. You then check to see if anything broke (no idea how long to wait, 1 day?). Then you increase the percent a little more (say, 1% to 5%) and wait a day again and check. Finally you update everyone (who has updates on)
At some scale such careless mistakes are going to create real effects for all users of internet through congestion as well.
If this was not a $8000 mistake but was somehow covered by a free tier or other plan from Google Cloud, would they still have considered it a serious bug and fixed it as promptly?
How many such poor designs are out there generating traffic and draining common resources.
Their users do not care about their screen recording studio anywhere near as much as the devs who wrote it do.
Once a month is probably plenty.
Personally, I disable auto-update on everything wherever possible, because the likelihood of annoying changes is much greater than welcome changes for almost all software I use, in my experience.
Screen Studio can collect basic usage data to help us improve the app, but you can opt out of it during the first launch. You can also opt out at any time in the app settings.
It is sort of fun (for $8,000) as it was “just” a screenshotter, but imagine this with bank app or any other heavily installed app.
All cloud providers should have alerts for excessive use of network by default. And they should ask developers if they really want to turn alerts off.
I remember Mapbox app that cost much more, just because provider did charge by months… and it was a great dispute who’s fault it was…
1) Emergency update for remote exploit fixes only
2) Regular updates
The emergency update can show a popup, but only once. It should explain the security risk. But allow user to decline, as you should never interrupt work in progress. After decline leave an always visible small warning banner in the app until approved.
The regular update should never popup, only show a very mild update reminder that is NOT always visible, instead behind a menu that is frequently used. Do not show notification badges, they frustrate people with inbox type 0 condition.
This is the most user friendly way of suggesting manual updates.
You have to understand, if user has 30 pieces of software, they have to update every day of the month. That is not a good overall user experience.
That's not an user issue tho, it's a "packaging and distribution of updates" issue which coincidentally has been solved for other OS:es using a package manager.
This could have easily been avoided by prompting the user for an update, not silently downloading it in the background... over and over.
If the update interval had been 1 day+, they probably wouldn't have noticed after one month when they had a 5 minute update check interval.
How the times have changed ..
If the file contains invalid JS (syntax error, or too new features for IE on Win7/8), or if it's >1MB (Chromium-based browsers & Electron limit), and the file is configured system-wide, then EVERY APP which uses wininet starts flooding the server with the requests over and over almost in an endless loop (missing/short error caching).
Over the years, this resulted in DDoSing my own server and blackholing its IP on BGP level (happened 10+ times), and after switching to public IPFS gateways to serve the files, Pinata IPFS gateway has blocked entire country, on IPFS.io gateway the files were in top #2 requests for weeks (impacting operational budget of the gateway).
All of the above happens with tight per-IP per-minute request limits and other measures to conserve the bandwidth. It's used by 500 000+ users daily. My web server is a $20/mo VPS with unmetered traffic, and thanks to this, I was never in the situation as the OP :)
The number of times I have caught junior or even experienced devs writing potential PII leaks is absolutely wild. It's just crazy easy in most systems to open yourself up to potential legal issues.
The website makes it seem like it's a one person shop.
If you're not confident you can review a piece of code you wrote and spot a potentially disastrous bug like the one in OP, write more tests.
Just amazed. Yea ‘write code carefully’ as if suggesting that’ll fix it is a rookie mistake.
So so frustrating when developers treat user machines like their test bed!
Avoidable, unfortunate, but the cost of slowing down development progress e.g. 10% is much higher.
But agree that senior gatekeepers should know by heart some places where review needs to be extra careful. Like security pitfalls, exponential fallback of error handling, and yeah, probably this.
After I shipped a bug the Director of Engineering told me I should "test better" (by clicking around the app). This was about 1 step away from "just don't write bugs" IMO.
TBH, that was well done for what it was but really called for automation and lacked unit-testing.
Although, after such a fuck up, I would be tempted to make a pre-release check that tests the compiled binary, not any unit test or whatever. Use LD_PRELOAD to hook the system timing functions(a quick google shows that libfaketime[0] exists, but I've never used it), launch the real program and speed up time to make sure it doesn't try to download more than once.
[0] https://github.com/wolfcw/libfaketime
I don't get the impression they did any testing at all.
In the grand scheme of things, $8k is not much money for a business, right? Like we can be pretty sure nobody at Google said “a-ha, if we don’t notify the users, we will be able sneak $8k out of their wallets at a time.” I think it is more likely that they don’t really care that much about this market, other than generally creating an environment where their products are well known.
The author seemed to enjoy calculating the massive bandwidth numbers, but didn’t stop to question whether 5 minutes was a totally ridiculous.
Well, you should hire contractor to set console for you.
"Designed for MacOS", aah don't worry, you will have the money from apes back in the no time. :)
Good on them. Most companies would cap their responsibility at a refund of their own service's fees, which is understandable as you can't really predict costs incurred by those using your service, but this is going above and beyond and it's great to see.
> Write your auto-updater code very carefully.
You have to be soooo careful with this stuff. Especially because your auto-updater code can brick your auto-updater.
It looks like they didn't do any testing of their auto update code at all, otherwise they would have caught it immediately.
Seriously this alone makes me question everything about this app.
https://en.m.wikipedia.org/wiki/Knight_Capital_Group#2012_st...
440m usd
The url specifically asks Wikipedia to serve the mobile site.
Previous discussion: https://news.ycombinator.com/item?id=35858778
I understand the reasoning, but that makes it feel a bit too close to a C&C server for my liking. If the update server ever gets compromised, I imagine this could increase the damage done drastically.
The title should have been: "how a single line of code cost our users probably more than $8000"
$229 per year on a closed source product and this is the level of quality you can expect.
You can have all the respect for users in the world, but if you write downright hazardous code then you're only doing them a disservice. What happened to all the metered internet plans you blasted for 3 months? Are you going to make those users whole?
Learning from and owning your mistake is great and all, but you shouldn't be proud or gloating about this in any way, shape, or form. It is a very awkward and disrespectful flex on your customers.
I think that is the essence of what is wrong with the cloud costs. Defaulting to possibility for everyone to scale rapidly while in reality 99% have quite predictable costs month over month.
What might be fun is figuring out all the ways this bug could have been avoided.
Another way to avoid this problem would have been using a form of “content addressable storage”. For those who are new, this is just a fancy way of saying make sure to store/distribute the hash (ex. Sha256) of what you’re distributing and store it on disk in a way that content can be effectively deduplicated by name.
It’s probably not so easy as to make it a rule, but most of the time, an update download should probably do this
The most obvious one is setting up billing alerts.
Past a certain level of complexity, you're often better off focusing on mitigation that trying to avoid every instance of a certain kind of error.
Electron really messed up a few things in this world
For those interested in this topic, and how other industries (e.g. Airline industry) deal with learning from or preventing failure: Sidney Dekker is the authority in this domain. Things like Restorative Just Culture, or Field guide to understanding human error could one day apply to our industry as well: https://sidneydekker.com/books.
Curious where the high-water mark is across all HNers (:
Our team had a bug that cost us about $120k over a week.
Another bug running on a large system had an unmeasurable cost. (Could $K, could be $M)
Set up daily emails.
Set up cost anomaly alerts.
I’m sorry but it’s exactly cases like these that should be covered by some kind of test, especially When diving into a refactor. Admittedly it’s nice to hear people share their mistakes and horror stories, I would get some stick for this at work.
The relevance is that instead of checking for a change every 5 minutes, the delay wasn't working at all, so the check ran as fast as possible in a tight loop. This was between a server and a blob storage account, so there was no network bottleneck to slow things down either.
It turns out that if you read a few megabytes 1,000 times per second all day, every day, those fractions of a cent per request are going to add up!
https://obsproject.com/