Saturday, August 31, 2013

Handling mongodb connections

Mongodb is fast and for medium to large setup, it just works out of the box. But for setups which are bigger than large, you may run into a situation where the number of connections max out. So for extra large setups, let us look at how to increase the number of connections in a mongodb server.

Mongodb uses file descriptors to manage connections. In most unix-like operating systems, the default number of file descriptors available are set to 1024. This can be verified by using the command ulimit -n which should give the output as 1024. Ulimit is the per user limitation of various resources. Ulimit can be temporarily changed by issueing the command ulimit -n .

To change file descriptors system wide, change or add the following line to your /etc/sysctl.conf

fs.file-max = 64000

Now every user in your server would be able to use 64000 file descriptors instead of the earlier 1024. For a per user configuration, you will have to tweak the hard and soft limits in /etc/security/limits.conf.

By default the mongodb configuration file mostly mongodb.conf does not specify the number of max connections. It depends directly on the number of available file descriptors. But you can control it using the maxConns variable. Suppose you want to set the number of max connections to 8000, you will have to put the following configuration line in your mongodb.conf file.

maxConns = 8000

Remember : mongodb cannot use more than 20,000 connections on a server.

I recently came across a scenario where my mongodb which was using 7000 connections maxed out. I have a replica set configured, where there is a single master and multiple replicas. All reads and writes were happening on the master. With replica sets, the problem is that if the master mongo is restarted, any one of the replica mongo servers may become the master or primary. The TMC problem was caused by a missing index which caused multiple update queries to get queued up. To solve the problem, firstly the missing index was applied. Next all queries which were locking the collection had to be killed.

Here is a quick function that we were able to put together to first list all pending write operations and then kill them.

db.currentOp().inprog.forEach(
   function(d){
     if(d.waitingForLock && d.lockType != "read")
        printjson(d.opid);       
        db.killOp(d.opid);       
        i++
     })

Chevrolet Aveo ownership report

In 2006, when I was looking for a sedan to purchase, I had multiple options. Ford fiesta and hyundai verna were recently released. Honda city was selling like hot cakes. I was opting for either ford or hyundai. But then while discussing my options around, I came to know the chevrolet aveo was also there in the same segment and has some features which really impressed me - like the ground clearance and the engine performance. Surprisingly as per the maintenance figures shared by all three of them the chevy was coming out to be cheaper to maintain. So I went ahead and got it.

For 4 years, I was extremely happy with the car's performance. I used to get good average of around 12-13 kmpl. Maintenance was every 5000 kms and it costed me only around 3000/- per service. The car was peppy and light to drive. My daily driving was around 30-40 kms, so it was much cheaper to maintain.

Then my first accident happened - I was standing still while a truck chewed through the driver side of my car. Driver side door and fender were totally damaged. When I went to the GM workshop, I could see the "smile" on their faces - which I was able to decrypt later. I got a quote of 60K from the GM workshop. I went to many other workshops, but they denied having parts for chevrolet cars. Finally I was able to find a workshop which did the repairing in around 20K - from which I had to pay around 10K.

This was the point where I should have realized that life would not be easy anymore with my aveo. The 25K service was very expensive - costed me around 15K. My shockers had leaked. I could not feel the difference in the ride quality but as per recommendation of GM, I got them changed. From then onwards, every service started costing 8-10K.

My driving went up from 30 km per day to 100 km per day. And I would finish off 5000 kms in less than 3 months. Another service would cost me 8-10K. So in addition to the rising petrol prices, I had to pay around 4K per month on the servicing of the car. The pervious shockers had a warranty of around 1 year and immediately after finishing 1 year the shockers gave away. Got them changed again - paying from my own pocket.

At 34K kms, the default goodyear tyres that came with the car gave away. Once I had 3 punchures on way to office on the same day. The next day, I took an off from office and got new tyres - again costed me 18K. Another hit was immediately after the 35K service. At around 37K, my batteries gave away. Interestingly there is a cut in the battery fitted in aveo, so only batteries from authorized workshops can be fitted. Also the complete computer would reset while fitting the battery and needs the authorization code to enable the car to be driven further. It is advanced technology but a pain. The battery which was normally costing around 6-7K in the market costed me 9K when I got it from the authorized workshop.

When my 40K service again costed me 10K for all the injector cleaning, oil change and other stuff, I decided to call it quits. I dumped the car at home and got another one for office commuting. Decided to go for a tata safari which is famous for some issues cropping up now and then. I thought that a chevy sitting at home which gets driven for around 200-300 kms in a month will be cheaper to maintain. But I was wrong.

After 41K kms, one fine day, I took the car to GM workshop with the problem that the car keeps on racing - engine RPM stays above 2000. They did a thorough cleanup of the car engine, charged me 7K for it and told me that the car's ECM has gone bad. For those who are not aware of the ECM, it is the computer sitting inside the car and it controlls almost everything in the car including fuel, acceleration and other stuff. It is supposed to be very rugged and can withstand extreme dust and temperatures which is always the case inside the car. It is said that if the ECM goes bad the car should not start at all. Surprisingly my car was starting and rolling but was rolling very fast. 

They told me that it would cost another 30K to replace my ECM - as electronics are not covered in car insurance. I raised the problem with GM india on facebook and they told me to take the car back. This time they did not charge me again but did some thorough examination of the car and again told me that ECM has to be replaced. They offered me a discount of 1000 on labour charges. I was furious. A car which has been driven only for 41,000 kms can develop a bad ECM. Fortunately one of my friends knew a 3rd party agent who dealt in such stuff. I was able to get the car working in 18K with the agent's help. And sold off the car as soon as it was healthy - before anything else goes wrong.

Today, in less than 2 years, my tata safari has done 40K kms. With service interval of 15K kms and service cost of around 5-6K it is a much cheaper car to maintain with lesser parts going bad.

The problem with GM cars is that they give you a happy feeling for the first 3 years with around 30-35,000 kms. After which the maintenance cost shoots up. After the free service from GM expires, and since there is no availability of parts in the market, you are solely dependent on the authorized service centers to cheat you or charge you whatever they like. And since the cars are not very reliable, the resale price is very less. After 5 years of driving a GM, you may realize that the money you spent in getting and maintaining the car was too much.

Ford also follows a similar structure where parts are not readily available outside the authorized service centers. But still the availability is much better than that of a GM. Also ford cars are more reliable. Hyundai, maruti, tata and mahindra can be easily maintained outside the authorized service centers. And they are much cheaper to maintain as parts and service centers are easily available.

But I would clearly stay away from GM and would advice the same.