Lambda Infinite Loop: How To Avoid It.
Disclaimer:
I wanted to share this experience so anyone reading this - especially beginner - can take a lesson from my mistake.
Some Background
Recently I decided to create a project for my portfolio with AWS and Next JS. The project is using the new Media Stream API where users can take photos or videos with their device camera. The technology is fairly new so it doesn't support all browsers yet.
Since I'm dealing with images and videos, I'm using AWS S3 and I also want to compress the images before storing them on S3. So I choose to use AWS Lambda function where when I upload the image, S3 will trigger the Lambda function and resize the image if it's larger than a certain pixel in width.
However the problem when taking an image with a camera that faces the user (front camera) is that the result is flipping. Meaning that your left will be on the right and your right on the left so I need to re-flipping it again.
...And This Is Where The Loop Hell Begin
I might write the wrong code in the function, but honestly, I didn't know that it would create an infinite loop. What I write is if the image width is bigger than a certain pixel, then the image will be resized, flipped, and saved... else the image will just be flipped and saved.
Nothing fancy, but this code is going to do an infinite loop. Until today I still can not fully understand how it will trigger the loop over and over again and cost me thousands of dollars. I thought that it would only run once when the image was uploaded to S3, triggered the Lambda function, and saved it. That's it. But boy I was wrong.
How To Avoid it?
There are a couple of ways to avoid this. I'm not an experienced developer regarding AWS services, but I found these options might avoid the infinite loop.
- Always Check your Lambda invocation. Whenever you change your function, always check its invocation and look at any irregularities. When the function invokes when you doing nothing then that is a sign that something may be off with your code.
- Set a budget alarm. You can set a budget alarm with your AWS account. I set my budget to 10 USD and they did email me when it got to 80% of my budget. Unfortunately, I didn't check my email on the weekend and it cost me a thousand dollars.
- If you put S3 as a trigger, make sure you activate the bucket versioning for the S3 bucket. It can prevent the function from writing over and calling itself and write it over again.
- Another way is to test it by mocking it. I try my code at my local computer and the code runs just fine, so in my case, this is not very helpful at all. I do like how firebase functions have emulators so a developer can test their functions locally before pushing them to the cloud.
- The last option is to entirely delete your Lambda function/s or other services that triggered it. I do this since I don't have any clue what's wrong. Forum thread here.
CONCLUSION
Please be careful when you are 'trying' something new. Always check your bill every day when you subscribe to a new service, even if you choose the free tier and already submitted your credit card with them, lastly don't forget to check your email every day too.
For everyone reading this, don't be afraid to keep exploring new technology, because there is a bunch of great technology out there that you should explore, but just keep in mind to always be careful and keep track of what you are doing.
My case is still in negotiation and investigation. I do receive some credit adjustments but it is still too much for me to pay. I'm still negotiating with the team. For now, I'm thinking of creating a tutorial in Udemy or maybe back to my old job, I don't really know what I'm going to do, just wish me luck.
That's it folks. Stay safe, and keep on learning!