Monday, January 14, 2013

Application recycling

For some time I was worried about unknown behaviour of my application. By analysing perfmon logs showing memory counters of iis processes and perfmon logs for operating system basic counters, I could clearly see that something bad is going on from time to time. Basically it looks like my application was being restarted. By looking at counters it looks like there is a memory leak in app (this is a very old app, and no one knows how it really works). The thing that I did not understand was why it is restarted, or what magic clears entire memory of app. Because I do not have an access to a production it was really hard to guess by looking at few perfmon logs. The next thing that I did not understand was why there was no error or warning in a log file (or event log in OS). As an addition my failover architecture was never triggered. Failover is set up to use a loadbalancer to ping my app and if it's not able to receive a response it redirects traffic to a different server. Finally it enlightens me that by default IIS is set up to recycle app every 1740 minutes (about 29 hours). When recycling happens, a new process is created in a background to handle new requests, and an old process is allowed to finish dealing with request that he was requested to deal with, before it is terminated – that’s why app is constantly responsive. As always the simplest answer is the hardest to think of.

No comments: