Saturday, December 30, 2006

mysql cluster

The NDB mysql cluster is an in-memory storage engine which offers high availability and data-persistence features. Mysql cluster is available with mysql-5.1 and with mysql5.0-max binaries. This can also be compiled after obtaining the source and the binaries can be used.

We will have a quick start to mysql ndb cluster here...
The basics first :

There are 3 nodes/services necessary to start when you are planning to run a mysql cluster

1. The NDB management node daemon ndb_mgmd.
2. The SQL node - a simple mysql server (mysqld).
3. Data nodes - running NDBD daemon

The best configuration would be to have at least 4 differet machines to run the mysql cluster on. The management node could be any small machine. All the management node does is check the cluster on periodic basis and other management functionalities like stop a node, start another node and stuff like that.

The Data nodes need to be high configuration machines - having lots of RAM. This is because the mysql cluster is a memory based cluster which means that the data resides in memory. The more memory you have the better the engine would perform.

And finally the SQL node should be a good machine since it will be doing some processing also. Though it should not require high amounts of RAM.

Steps to install and get mysql cluster running :

step 1:
install mysql 5.1 (binary) on 3 machines (2 data nodes and 1 sql node).
my.cnf on these nodes can be as below:

[mysqld]
ndbcluster
ndb-connectstring=ip.of.mgmnode

[mysql_cluster]
ndb-connectstring=ip.of.mgmnode

step 2:
untar the binary of mysql-5.1 and copy the files ndb_mgm & ndb_mgmd to /usr/local/bin
There is no need to keep the binary of mysql. Once the ndb_mgm* files are copied to bin directory, the untared binary of mysql can be deleted.

step 3:
configuring the mgm node.
create a directory "mkdir /var/lib/mysql-cluster".
create a file config.ini and incorporate mysql cluster settings in it

config.ini
[NDB DEFAULT] #affects settings of all data nodes
#No of replicas. Must divide evenly into the no of data nodes. So for 2 data nodes the no of replicas can be 1 or 2.
NoOfReplicas=2
# Memory allocated for data storage. Used for storing actual records and indexes. The memory space allocated consists of 32K pages. Once a page is allocated, it cannot be returned to the free pool, except by deleting the table. Default value is 80 MB. Minimum is 1 MB and max is dependent on available system memory.
DataMemory=1024M
# Memory allocated for Index storage (Hash indexes in mysql cluster). Hash indexes are used for primary key, unique indexes. Default value is 18 MB. Minimum is 1 MB and max is dependent on available system memory.
IndexMemory=128M
# Log levels. Used to log messages for various events. By default all loggings are disabled. For our reference we would set the log level to 5 for Startup messages and 10 for Error messages/warnings. Minimum level is 0 (disabled) and maximum is 15.
LogLevelStartup=5
LogLevelError=10

[TCP DEFAULT] # TCP/IP options
# no settings required for normal setup

[NDB_MGMD] # Options for the management process
hostname=ip.of.mgmnode
# Data directory - Directory where output files from the management server will be placed. It includes cluster log files, process output files and daemon's pid file.
datadir=/var/lib/mysql-cluster

[NDBD] # Options for data node "A"
hostname=ip.of.datanode_A
# datadir specifies the directory on data node "A" where trace, log, pid and error log files are placed
datadir=/usr/local/mysql/data # assuming the path on node "A"

[NDBD] # Options for data node "B"
hostname=ip.of.datanode_B
datadir=/usr/local/mysql/data # assuming the path on node "B"

[MYSQLD] # Options for the SQL node
hostname=ip.of.sqlnode

step 4:
Start the cluster. Hang on. The way to start the cluster is to first start the management node. Use the following command on management node machine.

ndb_mgmd -f /var/lib/mysql-cluster/config.ini

now run "ndb_mgm". It will connect to the mgm daemon on port 1186. use the "show" command to list the current settings.

ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 (not connected, accepting connect from ip.of.datanode_A)
id=3 (not connected, accepting connect from ip.of.datanode_B)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @ip.of.mgmnode (Version: 5.1.14)

[mysqld(API)] 1 node(s)
id=4 (not connected, accepting connect from ip.of.sqlnode)


Now start the data nodes. On the data nodes run the following command.

./ndbd --initial

The --initial command deletes all log files and clears the data on node machines. Use this parameter only for starting the data node for the first time.

And start the sql node like when you start a mysql server

./bin/mysqld_safe --user=mysql &

And run the "show" command on the mgm node again.

ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 @ip.of.datanode_A (Version: 5.1.14, Nodegroup: 0, Master)
id=3 @ip.of.datanode_B (Version: 5.1.14, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @ip.of.mgmnode (Version: 5.1.14)

[mysqld(API)] 1 node(s)
id=4 @ip.of.sqlnode (Version: 5.1.14)



Ok, now your mysql cluster is up and running.

If you need to restart the mysql cluster safely run the following commands

ndb_mgm -e shutdown # shuts down the mysql management and data nodes
ndb_mgmd -f /var/lib/mysql-cluster/config.ini #start the management node
ndbd # run this on the data nodes to start mysql cluster on data nodes
./bin/mysqld_safe --user=mysql & # And also shutdown and restart the sql node - like you shutdown and restart a normal mysql server.

To create a table using NDB cluster specify "ENGINE=NDBCLUSTER" in the create table syntax. Eg:

Create table testndb (id int, val varchar(255) ENGINE=NDBCLUSTER;

One error that i encountered while working with the NDB cluster is "ERROR 1114 (HY000): The table '#sql-15c1_2' is full". After a bit of digging i found the following in the log files on the mgm node.

2006-12-30 12:50:59 [MgmSrvr] INFO -- Node 3: Data usage increased to 80%(13205 32K pages of total 16384)
2006-12-30 12:50:59 [MgmSrvr] INFO -- Node 2: Data usage increased to 80%(13220 32K pages of total 16384)
2006-12-30 12:51:29 [MgmSrvr] INFO -- Node 3: Data usage increased to 90%(14757 32K pages of total 16384)
2006-12-30 12:51:29 [MgmSrvr] INFO -- Node 2: Data usage increased to 90%(14769 32K pages of total 16384)
2006-12-30 12:51:45 [MgmSrvr] INFO -- Node 2: Local checkpoint 12 started. Keep GCI = 168 oldest restorable GCI = 192
2006-12-30 12:52:13 [MgmSrvr] INFO -- Node 3: Data usage decreased to 86%(14112 32K pages of total 16384)
2006-12-30 12:52:13 [MgmSrvr] INFO -- Node 2: Data usage decreased to 86%(14109 32K pages of total 16384)
2006-12-30 12:52:17 [MgmSrvr] INFO -- Node 3: Data usage decreased to 74%(12212 32K pages of total 16384)
2006-12-30 12:52:17 [MgmSrvr] INFO -- Node 2: Data usage decreased to 74%(12207 32K pages of total 16384)
2006-12-30 12:52:33 [MgmSrvr] INFO -- Node 2: Local checkpoint 13 started. Keep GCI = 193 oldest restorable GCI = 192


I had allocated 512MB as the DataMemory, which was getting exhausted. So i increased the parameter to 1024M (1G) and since then the things are running fine.

The only problem i am facing is that I need huge amounts of RAM for storing the NDB tables. And i am looking for an option to reduce RAM usage and use simple data on disk clusters.

References:
Mysql Cluster Configuration
Mysql Cluster Complete documentation

Sunday, December 17, 2006

cars

Where do i start from?? Ummm, well, I have got a new car - a chevrolet aveo 1.6 LT (limited edition). Pic below...



The question is why did i get an aveo. Why not any other car? First of all, i never wanted to get a small car. I could have got a santro or a zen or an alto or a swift (i would have preferred the swift). But i wanted my car to be loooong. The options that i had were

1. honda city
2. chevrolet aveo
3. ford fiesta (petrol & diesel)
4. hyundai verna (petrol & diesel)
5. hyundai ascent (petrol & diesel)
6. Maruti Esteem
7. Maruti Baleno

If you look at both comfort and mileage and ease of maintenance, the best car is maruti baleno. But i dont like the looks of baleno. Ummm, let me give you some chart for comparison. Which is available from any other site (try carwale.com).

First of all, i was confused between getting a petrol or a diesel car. Though theoretically diesel cars are supposed to be much stronger and last longer(you can refer to my earlier posts for info on this.), but the initial cost and maintenance cost of the car is high. So unless and until you have to drive more than 100 kms in a day, it is not advisable to get a diesel car. The diesel car is approximately 1.2-1.5 lakhs costlier than its petrol counterpart and the maintenance of the diesel comes to be about 3000-5000 Rs extra per year.

If I would have gone for a diesel car, the cost of the fully loaded car would go beyond my budget. I liked the fiesta diesel and the verna diesel. Though the turbo lag is very much pronounced in the verna, but the pickup and power of verna is too good. Since my daily run would not be more than 50 Kms, i decided to go for a petrol car. I could get a fully loaded petrol car for around 7.6 Lakhs. Where as if i get a fully loaded diesel car, it would cost me around 9 lakhs.

This narrowed down my list of options to

1. hyundai ascent viva crdi
2. hyundai verna xxi
3. ford fiesta 1.6 sxi
4. maruti baleno
5. honda city
6. Chevrolet aveo 1.6 LT

Again knocking out what i did not like. Hyundai ascent viva crdi - wonderful car at reasonable price. I could have got this car for 7.2 lakhs and saved on petrol also. But the point was that the crdi technology in this car is somewhat old.

Well, I dont like the "ass" of baleno, so that also gets knocked out. But baleno has the best average among all cars. Baleno would give you an average of 14 in city and 17 on highway. And the interiors of baleno is very classic.

So now i am left with 4 options. And this is the comparison chart i made.

















CarFord FiestaHonda CityHyundai vernaChevrolet Aveo
Price7,65,000N/A (but 8L approx.)N/A (but 7.2L approx.)7,00,000
Engine TechnologyDurateci-DSIVTVT (Variable timing valve train)VGIS (Variable Geometry intake system)
Engine capacity (cc)1596149715991598
Power (PS@RPM)101@650077@5000103@5500102@6000
Torque (NM@RPM)146@3400125@2700146@3000147@3500
Fuel systemSEFIi-DSIN/ASequencial Fuel Injection with 32 bit ECM
Turning radius (m)4.94.955
Average (kmpl)11121111
Length (mm)4282439041304310
Width (mm)1686169516951710
Height (mm)1468149514901505
Wheel base (mm)2486245025002480
Ground clearance (mm)168160170181
Music system6 CD player with RADIO (no Mp3)Radio + CD/MP3 player with remoteRadio+mp3+cd player (4 speakers)Alpine music system with CD+MP3+Radio (6 speakers)


After going through all the options thoroughly, i zeroed down on the aveo. Well why not fiesta - coz it is costly to maintain. Another point was the music system. The music system in fiesta is damn good, but it cannot play mp3s. Imagine creating Audio cds - 6 of them to play in car and they would carry around 60, maybe 70 songs. Whereas one mp3 cd carries more than a 100 mp3 songs of really good quality. And I can cut one whenever i like. And the biggest point is that the ground clearance of aveo is very good. If you drive a fiesta on a road with potholes and mountains, you can expect some scratching on its base. But with aveo, the chances are grossly reduced. Aveo has got all wheel independent suspension - much better to drive on indian roads. And the best part is that it is cheaper than the other cars.

The city is a piece of trash - no power all style. Verna is a also a good car, but the diesel version is a lot better than the petrol one.

Lets see how things go...

Friday, December 15, 2006

insomnia

insomnia is defined as inability to sleep or take rest.

Here i am referring to the nights we spend in office working. How many days+nights would you have worked in a single go? I have done 4days+nights without sleep. And i believe there are people who have done a lot more.

It is not that I am workaholic. No dear, it is just that the work needs to be completed on time and when deadlines approach the efficiency of the human brain also increases and so does the period during which the brain will be active. The human brain is a wonderful thing. During emergency situations both efficiency and activity of the human brain increases by a good factor.

Oooops!, i think i am going off topic. This post is to discuss what you feel and how you feel when you have not slept for some time.

Lets start with the situation first. I am not a script writer, so i hope this turns out to be fine...

Act 1, Scene 1 - project is delayed and need to be deployed tonight. We have not done proper planning for the deployment.

Act 2, Scene 2 - It is 10'o clock in the night and we are terribly hungry. We need to get some food into our stomach and also start the actual deployment

Act 3, Scene 3 - We have had a heavy diner (sponsored by the company) and it is 11'o clock. Our stomach is full and our eyes are drowsy. And so now the deployment starts.

Act 4, Scene 4 - We start cracking senseless jokes. And to add on it we start laughing on those senseless jokes. We are drowsy. Simply copy pasting commands from the deployment doc to the system. Brain utility is at the minimum. And we are having coffee after coffee to keep us awake.

Act 5, Scene 5 - The project is deployed and all are happy. Now we have started testing the project to see if we have missed anything. Boooom!! a flaw is found, second flaw is found, third..... We are now wide awake. The brain is fully active. Heartbeats are up. The heart is pumping more blood to the brain. We fight. We debug and decode our own scripts. And finally after much struggle around 4 in the night we think we are done.

Act 6, Scene 6 - Now we need to decide whether we should go to sleep and return early to handle problems or go to some place, party and be back in the morning without sleeping. By this time your eyes are closing. You feel a bit heavy. Your food has still not gone down. You need to drink a lot of water. After a lot of struggle and arguments, we decide that a 2 hour sleep would do some good. So we rush home and fall on our beds.

Act 7, Scene 7 - It is 7 in the morning. And the maid is downstairs ringing the bell for some time now. She is paitent. I have woken up, but dont want to get out of the quilt. Wishing that she gives up and goes away. But she is very insistent. Why does she has to clean the room today. Cant she take rest for a day. Well, finally i get up with around 2-2.5 hours of sleep, open the door and let her in. And then jump back in bed hoping to catch 30 minutes of sleep more. But the maid has her own set of questions. Where are the clothes to be washed? Why have we not got the resources (bucket, soap, etc) to aid her cleaning process? I just wish she would do her work quietly and shut the door when she goes. But nooooo. Well, I answer her - "will get the resources today, etc.". Just when the maid is about to go and i am dreaming about my peaceful sleep, the bell is rung again. The car-wash person is here and wants me to open up the car so that he can clean it from inside. I suddenly have a notion that the world is very dirty today morning. And all cleaning needs to be done today itself. Why dont people simply stop cleaning everything? Why is there so much dust in this world?

Act 8, Scene 8 - So I am now fully awake. The car-wash person has gone and my car is gleaming bright red. I can feel that my brain is tired but I know that no matter how much i try, i would not be able to sleep now. Well anyways, I take a bath to freshen up and shoo away the sleep. But the bath does not help in any way - except increase the effort in changing clothes. I feel as if I am a day old. I am hungry but my stomach is full. Anyways, i start my bike and drive to office. I feel as if there is a lag between the situations i see and the reactions which i am supposed to do.

Act 9, Scene 9 - I am in office, hungry and thirsty. Park my bike, punch my card. Plugin my lappy and look at the mails. The project deployed has bug reports already. Good!!! Start the debugging process. Drink a lot of water to keep myself awake and make a few no of trips to the toilet. Take coffee.

Act 10, Scene 10 - It is 12 am. My body has started reacting to sleeplessness. The mind has somehow communicated the lack of sleep to the body. The body now has a tingling sensation. Stomach is still full and hunger is still there. The hunger is becoming more prominent. Eyes are drowsy. Crack pot jokes have gone down. People came and went with their "good work" slogans. I just wish they had the courage enough to stay up a few nights so that they realize our "sacrifice".

Act 11, Scene 11 - It is 1:26PM and it is lunch time. Finally i am going to get some food to eat. But I am not sure where i am going to put it as my stomach is still full. Tiredness has increased. Stuff myself up with whatever junk and stale food that was available.

Act 12, Scene 12 - It is 2:18 PM and i am feeling very very sleepy. My eye lids are not agreeing the command that i am giving and are trying their best to close my eyes. Suddenly i am drowsing on the chair itself. My hand is on the keyboard, eyes half open - half closed. And this is just the moment when my boss makes his appearance - bad luck and wakes me up. All work gone to drain. Get calls from banks offering some stupid credit card. Just want to sleep. My mouth is dry. The uncomfortable chair that i sit upon has now become very comfortable. If i leave before 4:30, it will be considered as a half day. So i will have to be in office and be unproductive till 4:30.

Act 13, Scene 13 - It is 4:15 and i cant take it any more. My eyes are by default half closed. And i can fall sleep walking. Dont remember how much water have i drunk. Just remember that i have been to the loo lots of times. My stomach has suddenly grown huge and full. I dont feel hunger any more - just the need to sleep. I pick up my lappy and am back driving home. I dont care about the half day i would get due to my going home 15 minutes before time.

Act 14, Scene 14 - I am back home. Wanting to sleep. Buy my screwed up brain wont allow me to sleep now. I am on my bed with my lappy and watching movies - reading novels. I realize that there are stretches of time when i have not read anything or saw the movie. I have missed an entire scene and i dont remember what i have done during that time. I had not slept. And i had not pondered on any issue. Have not thought any thought. I was just blank...

Act 15, Scene 15 - It is 8 PM and i dont feel hungry any more. I dont feel thirsty any more. I dont feel the need to watch a movie. I dont feel the need to sleep also. My mind is still tired and my stomach is still full. My mouth is still dry. And now i can meditate - if that is what you call when a person can remain blank and without feeling anything around him. I am at peace.

Act 16, Scene 16 - It is 10 PM and i am again stuffed with junk food. I am on my bed and sleeping very heavily. I dont dream anything. I dont feel anything. You can cut me in my sleep and i wont feel you cutting me.

I wish that i can sleep like this every night.

P.S. =>> Some scenes have been tampered with (a little) to make the story more interesting and readable.

Thursday, December 07, 2006

sociallization

The question here is how do people manage their social life... Well, frankly, i am tired...

To start off, i am engaged, have a 9 to 6 office, have to prepare for my marriage, have to get a house and a car and prepare for my coming married life. And i am not sure how to get all these things running in one go.

Have to talk to my mom, have to talk to my sisters. Make small talks. I dont know why, but i find it difficult to make small talk. According to me, people should just talk when they have something to convey, something to discuss or something to get done.

I get up at 7 in the morning, the maid comes and starts ringing the bell. She has to be let in so she can start doing her household works. Then my fiance would buzz me in the morning to wake me up - so sweet of her. I like this about her. I get ready and reach office by 9:30. And then there is work. Mails to respond to, meetings to attend and issues to solve.

In the evening, i already have a list of todos which need to be done. And of course go for food. Talk to mom, discuss her issues. Talk to the carwala and blast him off. On top of it people complain that i dont interact with them. People get cross and then again i have to work on it.

I dont know how people tend to live in joint families. I dont even tend to remember the names of all my relatives and neither do i remember their relations with me...

When should i get my car serviced and when should i get the water purifier installed. Well, thats comparatively easy. When should i call people who expect to be in touch with me and what small talk should i discuss with them. I have nothing to say to them.

And ofcourse, my friends - some freaking out/ hanging out in the night. Watch a movie - go and explore a new place.

I just dont get it. How do people manage it, keeping in touch with all the people you know.

Maybe sometime down the line, i will be able to do all that without a hitch. The problem i think is that i intend to live up to everybody's expectations.

How to people manage to keep more than 1 girl friends? Handling even 1 girl is tiring...

Whoooooooooo, lets c where i go from here....