Adding 5 line of log caused us to have severe memory leak in our NodeJS service

It was suppose to be a smooth production deploy, of a NodeJS micro service that was updated to have 5 more lines of logs.

Before deploying the new container the following happened

1. Local UT passed
2. Exploration test passed - showing the new logs and they aced
3. CI passed and a new docker was created to ensure no funky stuff

We deployed at 11:30 AM and all was cool and happy.

At 7:30 PM we got a phone call from our amazing DevOps team (who is now recruiting... PM me if you are in Israel!) that the service we deployed crashed with OOM

The info from the failure shows "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory"

Our NodeJS service use the default heap of 1.73G.

We wanted to take 2 heap dump so that we can compare , there are several ways to do so:

1. Add heapdump package, which allows you to create heap dump to a file with a signal from outside
2. Live debugging session (running node with --inspect OR simply signaling for immediate debug)

We took 2 dumps and used the chrome://inspect to load them and compare,

The outcome was pretty slick - we immediately identified a potential leak and when checking the module github repo, we identified an open defect

The immediate solution was to ensure we use previous version so that we can deploy new version.

Several things to remember:

1. If you don't lock dependencies - you are bound to get defects. I am not saying you need but just be aware
2. When you run NPM INSTALL locally, unless the package.json policy was changed to require new versions, your current one will always be sufficient and will never get the latest
3. It's highly suggested to add isolated performance test for your micro service - mock all external calls and ensure you service doesn't leak / behave correctly 

Check out the video (no audio of how we did it):