LiveJournal - system architecture

Lets discuss the system architecture of Live Journal.

Live Journal or LJ for short kicked off as a hobby project in April 1999 and was built on open source completely. It reached 2.8 Million accounts in April 2004 and 6.8 Million accounts in April 2005. Currently It has more than 10 Million accounts. Caters to several thousands of hits per second and lots of MySQL queries.

Here is a complex diagram which roughly outlines the architecture of LJ.

The technologies which are visible over here are -

Caching - Memcached
Mysql Clusters - HA & LB
Httpd load balancing - using perlbal
MogileFS - Distributed File System

Lets start off with mysql...

A single server with mysql wont be able to handle the large no of reads and writes. With increasing no of reads and writes, the server slows down. Next stage would be to have 2 servers in a master-slave architecture in which the master handles all inserts and the slaves are read-only. But then the queries have to be spread over in such a manner that replication lag between master and slave (though very small) is handled. As the number of database and web servers are increased - chaos increases. Site is fast for a while and then again slow - and there is need for more servers with higher configurations. Also, as the number of slaves increases, the number of writes to the slave also increases. So eventually you come to a situation where the number of writes is very large as compared to the number of reads. Resulting in large I/O and low CPU utilization.

The best way to handle such situation is to divide the database. How LJ did this was by creating user clusters. So each user was assigned a cluster number. And each cluster had multiple machines in a master-slave fashion. The first query would then find the cluster number for that user from the global database and subsequent queries for that user could then be redirected to the user cluster. Ofcourse few issues like uniqueness of userid, and moving user around clusters had to be tackled. Caching of mysql connections and using mysql query cache to cache query results added to the better performance of the site.

Again the problem was the single point of failure with the master databases. If any of the master database dies, the site would go down. To avoid this situation master-master cluster was created. In case of any problem - the other master would come into play and handle all active connections.

Which database engine to use - InnoDB or MyISAM. InnoDB allows concurrent reads and writes and so is comparatively fast. Whereas MyISAM has table level locks and so is not as fast as InnoDB.

And then there is MySQL cluster which is an in-memory engine. It requires about 2-4x of RAM for the dataset. So it is good for handling small data sets only.

An even better way of storing database is by using shared storage - SAN, SCSI, DRDB. You turn a pair of InnoDB machines to a cluster - looks like a single box from outside with floating IP address. Heartbeat to move IP, mount/unmount filesystem, start/stop mysql. DRDB can be used to sync one machine's block device with another. This requires dedicated gigabit cable between the two machines to handle the high amount of data transfer.

Cache

Memcache is used to cache records which has already been computed for frequent access. Memcache is an open source distributed caching system - instances of which can be run on any machine where-ever free memory is available. It also provides simple APIs for different languages like java, php, perl, python and ruby. And it is extremely fast.

LJ created 28 instances of memcache on 12 machines (not dedicated) and was able to cache 30 GB of data. This cache was getting a hit rate of 90-93%. Which reduced the number of queries to the database to a great extent. They started caching stuff which was very frequently accessed and aim at caching almost everything possible. With cache - there is an extra overhead of updating the cache.

http load balancing

After trying a large number of reverse proxies, LJ people were unable to find anything which satisfied their needs. So they built up their own reverse proxy - perlbal - a small, fast, manageable, HTTP web server which can do internal redirects.
It is single threaded, asynchronous and event based. Handles dead nodes. And works in multiple modes - static web server, reverse proxy and plug-ins.

Allows persistent connections and has no complex load balancing logic - uses whatever is free. Connects fast and has multiple queues - for free and paid users.


MogileFS - Distributed File System

Files belong to classes. It tracks what disks are files on. Keeps replicas on devices on different hosts. It has libraries available for most of the languages - php, perl, java, python.

clients, trackers, mysql database cluster and storage nodes - all were brought under MogileFS. It handles automatic file replication, deletion etc.

Have put in only major points and finer details can be found in the link below.


source:
http://danga.com/words/2005_oscon/oscon-2005.pdf

realtime fulltext index

For past some days, i have been struggling with getting something which can allow me to create, insert, update and search fulltext indexes - big fulltext indexes. Not very large but somewhere around 3-4 GB of data.

You would suggest mysql, but with mysql the inserts on table with fulltext indexes is very slow. For every insert, data is put in the index which leads to slow inserts.

What else? Well, i had come across some other fulltext engines like sphinx and senna. Compiled mysql with senna but there was no benefit. The index size was almost double and the searches were also slow - as slow as mysql fulltext index.

What about sphinx. Well, had integrated sphinx with mysql. But the sphinx engine has lots of limitations - like the first 2 columns must be integer and the 3rd column should be a text. Rest all columns should be integers. So if i use sphinx, i would need to write a stored procedure and trigger it to port data from my current table to a parallel sphinx table whenever a new row is inserted. And what would i do if i have to run a query - i would be joining both the tables. Would that be fast. Dont know. Let me figure out how to get this thing running...

You would say - what about lucene - my favorite search engine. Well dear, lucene is in java and there is no way i could integrate it with mysql if i have to do it in a jiffy. I would need to design and build a library to get this thing running. Forget it. In fact dbsight provides a similar type of library. Maybe i will get that thing working and check it out. At a small price, I might get what the organization requires and that too without wasting any resources on building it.

Hope i get some solution to the situation i am in right now. I have solutions for it, but not a solution which requires least amount of resources to get it running. Till i get one, the search will continue...

water bridge

Water Bridge in Germany .... What a feat!
Six years, 500 million euros, 918 meters long.......now this is engineering!
This is a channel-bridge over the River Elbe and joins the former East and West Germany, as part of the unification project. It is located in the city of Magdeburg, near Berlin. The photo was taken on the day of inauguration.

corruption - part dwitia

It has been a long time since i have written anything - either technical or on any other topic. Hmm... Well, i was busy getting married....

I have just joined my office - a week back and am getting in the flow of going to work.

Had been on honeymoon and have been travelling. Have seen the different ways corruption works in india. Each and every incident involving corruption by government officials is worth noting. But right now i will list down some incidents that happened with me.

Incident #1
I am coming from sector 18 towards sector 16 in noida. I am on a red light and i dont see a "thulla" (in india all police officials are called thulla. They are always ready to take money from you and fill their own pockets) around. Well anyways, i still decide to wait till the light turns green. There is an auto coming from the left and he is also waiting for his signal to turn green. Still, every minute the auto moves 1 meter ahead. My red-light turns green and by that time, the auto is almost in the middle of the road - and there is still no thulla around. No one paying attention to him. So i start moving and am almost half way to the across when the auto decides to move still ahead and comes right in front of me. I stop and before i can turn and head across the crossing, my signal turns red.

The autowala gives me a frown and blows his horn. And from no-where a thulla jumps out and signals me to stop. Well, i shud have sped off. But, my mistake, i decide to stop and make him understand the situation. I am in hurry. I dont keep more than 400 in my pocket. The thulla asks me for my papers. I tell him that i dont have any. And that i live nearby - just across the road (I told him the truth). He tells me that red light jumping is 1000 Rs fine. And then sits behind me and asks me to ride to the nearby police station. And one most important point - he is not a traffic police. So i tell him that i am not driving to the police station. I dont have 1000 Rs fine and try to make him understand the situation. But he refuses to listen and tells me that he can let me go with around 400/- cash to him. I shut off my bike. Took the keys out. Hand it over to him and tell him to take my bike to the police station and i will collect it from there after paying the fine. The thulla just looks at me and lowers his offer to let me go to 300 Rs.

I argue with him - give him all the lies i can make up. Make him understand that i am a poor student and that i dont have enough money in my pocket. Eat his head for around 1 hour. The thulla is fed up and starts calling a crand to take by bike. I asked him "why dont you take it yourself?" And he tells me that he does not know how to drive a bullet. Shame on him. Finally people start gathering and looking at us. He starts shouting at them. I argue with him openly and tell him that 50 Rs is enough. He is still adamant and we finally settle for 100 Rs. I hand him my hard earned 100 Rs for something i have not done and he goes away happy.


Incident #2
This is a really nice one. We (me and my roomie deepak) used to live in noida sector 12 and we used to go to vasundra enclave for our diner. There is a small resturant over there which serves good food at low prices. And the best part of the resturant was that they did not levy sales tax of 12.whatever percent.

So one night after having a lavish diner, we were riding back on deepak's scooty. Both of us wearing helmet - cause there are thullas always ready at the border to extort money from you. Still, when we were crossing the border, a junior thulla stopped us. And asked us for the papers. Hmm and my romie depak had forgotten his wallet home. So he checked everywhere and finally told that he has forgotten his wallet home and he would need to go back home which was nearby - around 5 minutes from there to get his wallet and papers. This was reported to the senior thulla and he came and told us that if we dont have the papers, then we should pay the fine.

I thought that i would ask the question - fine for what. Well, but i didnt. It is better not to provoke a thulla. I told deepak - you wait here and i will get your papers from home. So i took a rickshaw and moved off towards home to get the papers. I had just reached home, when i got a call from deepak that the thulla has let him go and he is riding back home. And then he told me what happened.

The senior thulla told the junior thulla "dekh kar roko. Jinke pas paisa nahi hai unko rok kar kya faida". [look at the person before you ask him to stop. There is no point in stopping someone if he does not have any money]. Deepak was simply shood away - since he did not have any money on him.


Incident #3
This one is a very recent one - happened in gurgao while I had been to gurgaon to pick up my wife.

I started early from my office so i could reach her office on time. And when i reached there - there was a huge traffic jam outside her office. Inspite of that there were cars parked outside her office on one side of the road and there was a thulla looking around. It didnt seem wrong to me to stop on the side of the road and wait for her to come down. Stopping for 5-10 minutes should not be an issue. If the thulla thinks it is an issue, he can tell me and I can simply drive off.

So i have parked and put on my blinkers and am waiting when this thulla comes and knocks on the window. My mistake, i should have simply drove off. I open up the window and the thulla asks me for the papers. I tell him that i will move on if there is a problem. But he is adamant on getting the papers and wants to pose a fine of 400. Finally after some argument, he takes off my car keys.

I go to him with my papers and explain him that if he takes the papers then i will have to drive back from noida to here to collect my papers. So take the money now and let me off. And the thulla simply says that if you give the money now, it will go into my pocket. I tell him - then i could be led off at a much lower amount. He asks 300. I empty my pockets and give him all 10s and 20s - totalling around 100. The thulla says that there is an ATM in the nearby building, get the money from there. So i go and get 300 for him and hand it over to him and get my papers and drive off.


There are so many incidents with me - all almost similar. Maybe i get trapped cause i stop when the thullas ask me to stop.

I have heard incidents from my friends where one guy said that he told that he wont give the money now and would go to court. And he was disappointed cause there is no one to listen in court. All they assume is that we are murderers or maybe somewhat similar and we should pay double or tripple the amount fined on us. So it is better to not to go to court.

One really nice incident happened with one of my friend. He was caught jumping a red light and he paid the fine. He says that he emptied his heart out. Gave all the bad words he had ever came across to all the thullas there. Called them the worst things he could call. And then paid his fine and drove off. And he was happy. He says that it was worth paying the fine. Letting go of this frustration and driving off.

The best thing about corruption is that the corrupt officials wont be paying any tax on their "under-table" income. Maybe government should put tax on all such incomes. Or maybe make money paid as corruption - exempt from tax. If i would have earned 100 Rs, i would have paid 30 Rs as tax and would have got to keep only 70 Rs with me. The under priviledged common man.

Now the government is increasing the fines for all the penalties that tend to happen. Indirectly raising the cut for all the thullas. I could now-a-days see the thullas drooling at the sight of people driving. And their drool increases with increase in the number of vehicles and the raise in penalties.

The problem is not with the corrupt police officials, but with the management/the government. They have created situations so that corruption prevails. There is no proper management of traffic signal timings. The thullas are underpaid - i think so. And the thulla will always catch you if you start crossing the traffic signal at green and arrive at the other end when the signal is red. And this usually happens cause the time for yellow light is so low that there is almost no yellow light. And the red light does not work for cycle or rickshaw. So cycle or rickshaw can jump as many red lights as possible and create as much confusion as possible on the road.

After all, this is INDIA...

world without engineers

Mechanical Engineer

Aeronautical Engineers

civil engineers

communication engineers

computer engineers

Electronics Engineer

world's best public toilet

This is a public toilet found in houston.

The outside




And the inside...




Well, so, it would ofcourse make shitting over here a memoriable experience...

ilaka triund dharamshala trip

That is a trip we went for during 26 jan 2007. I am terribly sorry for the delay in writing this post. I got a bit lazy. But now since i have nothing else to do, i might as well write the post. This is one of best trips i had ever had.

So, it starts on the very fine evening of 24th. We hire a qualis which is supposed to arrive at 7 pm. All our bags are packed and ready. And as per IST(Indian Stretchable Timings) we get a travera which arrives at 9:30 pm. And to top it up, the travera does not have a place to put our bags (no overhead luggage carrier). Though our luggage was minimal, we all 6 people being guys (gals generally tend to carry more luggage). So we push bags under the seats, keep some on the back seat and finally get packed in the travera. And we start our journey.

Within half an hour the sardarji, our driver reaches the bypass, and starts sleeping. Well not actually sleeping comfortably, but taking small naps during driving. And then we come to realize that the sardarji has not slept in 36 hours. Too bad. Well, so we offered him rest and one of us takes the wheels. We drove for about 5 hours. And then the sardarji takes the wheels again.

Cool, so around 4 in the morning we reach this place - a petrol pump from where we are supposed to start climbing uphill. And the sardarji just drives to the side of the road, pulls a sheet over him and without any word, just goes to sleep. And we all are looking at him with our eyes wide open. Well, we sleep for about 2 hours, and then continue our drive. We pass dharamshala and reach mclaudganj around 10 in the morning.

Here we took 2 rooms and got fresh and had breakfast, and then we start our climb uphill around 1 pm. The trek was good, and we took lots of snaps on the way. We saw a forest fire on the way. And by 5:30-6 pm we are almost on the top of the hill. We start seeing snow on our way. Snow which is still melting. Someway in middle we stopped and sat down and we could hear all the snow around the hill melting with a distinct sound. There was no other sound - except for the snow melting. Too good.

And around 6:30 we reach the top of the hill. It is a nice pleatu. With a population of us 6 people, 4 helpers, 6 horses, 1 caretaker of a guest house (There is a guest house on top of the hill and a caretaker to take care of it), and a dog. That was the population of triund when we reached there. On both sides we have deep valleys. One valley leading to the city. And other valley leading to hills covered with snow. You can have a look at the video below to get an idea of the beauty of the place. I dont have words to describe it.





So we had our diner that day and slept in sleeping bags in a tent on top of the hill. At first, i was a bit afraid that the tent might not be able to protect us from the cold. But the sleeping bags and tent were sufficient to protect us from the cold. Next morning we saw the sunrise from behind the snow covered mountains.

And around 11 started our trek to ilaka (snow line) where we were supposed to see pure snow. The path was very dangerous. We came across small paths covered with snow and slippery and had to trek through that. We were so terrified we were going to go back. But then the guide helped us out. The trek was such that one wrong foot/ one slip and we would have been at the bottom of the valley.



We reached the snow line and had a gala of a time. We played in snow and made snow man. The snow used to melt and get through our clothes. Our fingers were numb. The snow used to get though our shoes and make our feet cold.





We had our lunch on the top of the hill and in middle of snow. We relaxed a little after lunch. Enjoyed the sun and the cool breeze and the quietness. And we saw snowfall on our way back from ilaka.



In the evening after we reached, we saw that one of the shops (there are 3 shops in triund - the local people carry stuff uphill for tourists and sell it at huge profit margins) was open and had coffee there. In the night just before diner, we shouted with our hearts out. Our guide played flute and we made a small camp fire (though campfire is not allowed there).

You should hear this, it might be the best sound album created till date...



And we slept that night. Next morning we had breakfast and treked back from a different and more dangerous route. The route was very steep and at some times we were unable to find the path. The wind was wonderful.







After reaching the city, we tried some tibetian food and some shopping. But we were so tired and exhausted that we found nothing interesting. Anyways, in the end we again packed ourselves in the travera and drove back. Hoping that our adventure was over. But we were wrong. Our sardarji driver was again falling asleep on the way back and so he stopped at some place on the highway - we dont know where and took some drugs (afeem, i think) so that he stays awake throughout the night.

Well man that was the best drive i ever had. The guy was a super hero after he took the drugs. He drove through blinding fog at 80 kmph and then at an average of 110. We covered 220 kms in 2 hours.

At 4 in the morning, we reached home and slept....

hmmmmm bike

This is really innovative and fundooo..
Imagine wondering on this bike through the city, and that too at night...








Koooool....

my new car...

Sorry for the delay in pics...
But they are here and they are from my new camera.

The problem is that i cannot take a pic of my camera with my camera. So i would never be able to post the pic of my camera...

He he, Anyways. Have got a Canon powershot A710 IS - the camera I mean. The car here is Aveo...

memcache

Something new, something exciting, something which makes your brain churn... Thats what this blog is all about...

Hmm, so what are we to explore today...

Every machine has some amount of RAM where the OS/programs store frequently accessed data. Have you ever tried storing arrays in memory so that you can access it very frequently. You might have. Even i have done a similar stuff. Storing a binary tree in memory as doubly linked list - so that parsing of the tree becomes fast.

Here is something to ponder over known as memcache. Defined as "A high performance, distributed memory object caching system". You can get it here http://www.danga.com/memcached/

How does this work.. Well, firstly just download the tar.gz source file. Untar and compile it. A simple ./configure , make and make install would do.

So you will have the "memcached" binary ready. Then all you have to do is run the memcached binary in daemon mode and assign it some amount of memory where it can store data. Just do a memcached -h and it will list you all the available options

[jayant@jayant memcached-1.2.1]$ ./memcached -h
memcached 1.2.1
-p TCP port number to listen on (default: 11211)
-U UDP port number to listen on (default: 0, off)
-s unix socket path to listen on (disables network support)
-l interface to listen on, default is INDRR_ANY
-d run as a daemon
-r maximize core file limit
-u assume identity of (only when run as root)
-m max memory to use for items in megabytes, default is 64 MB
-M return error on memory exhausted (rather than removing items)
-c max simultaneous connections, default is 1024
-k lock down all paged memory
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-h print this help and exit
-i print memcached and libevent license
-b run a managed instanced (mnemonic: buckets)
-P save PID in , only used with -d option
-f chunk size growth factor, default 1.25
-n minimum space allocated for key+value+flags, default 48


To start memcached in daemon mode with 128 MB RAM, which listens on localhost port 11211 the following command would need to be run

memcached -d -m 128 -l 127.0.0.1 -p 11211

using the options available above, you can configure memcached as per your needs.

So now the server is up and running and you need to use clients to connect and store data over there. For that there are apis available with different languages which allow you to connect to memcached daemon and store/retrieve variables, arrays and objects from it. APIs for perl, python, ruby, java, C# and C are available on the website.

Since i generally do work on php, i wanted an API for php. Well for that the simplest way to install a memcache API for php is by running the following command as root

pecl install memcache

It will automatically download, compile and install the memcache API for php.

Cool, so now we are ready. We have the server running and the client API ready. All we need to do now is build a program which puts and gets information from memcache. I will stick to php for this. http://in.php.net/manual/en/ref.memcache.php lists the functions available with memcached. I wont be giving you a detailed program on how to use memcached api. But just for an idea sake...

You will have to connect to the memcached daemon using the memcache_connect function and then use the memcache_add, memcache_get, memcache_set and memcache_delete to add, retrieve, update and delete objects from the memcachd daemon.

Points to ponder upon:

1. Memcache is very fast. It uses the libevent to scale to any number of open connections.
2. You can start any number of memcached servers on different machines. Different instances of the server do not replicate data across each other. The client api uses the different servers to create a pool and distribute and store data among them.
3. LiveJournal.com uses memcache on a major basis for serving dynamic pages. It helped them reduce the page load times and also reduce database load by a major extent.
4. Memcache does not allow you to store objects which are language independent. So if you use php API to store an object, you cannot use java API to extract the object. Well, even if you extract it, you wont be able to figure out the object - it would be incomprehensible.

If you can figure out a way to serialize all objects of different languages in a well defined manner, then the same object could be stored and accessed by different languages from memcached. This should make things much simpler.