Networking Expert...
Programmers...
Thursday, October 23, 2008
Saturday, October 18, 2008
setting up hadoop
Have updated the post with latest hadoop config changes compatible with hadoop 1.1.2
Hadoop is a distributes file system similar to google file system. It uses map-reduce to process large amounts of data on a large number of nodes. I will give a brief step by step process to set up hadoop on single and multiple nodes.
First lets go with a single node:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
in hdfs-site.xml
Hadoop is a distributes file system similar to google file system. It uses map-reduce to process large amounts of data on a large number of nodes. I will give a brief step by step process to set up hadoop on single and multiple nodes.
First lets go with a single node:
- Download hadoop.tar.gz from hadoop.apache.org.
- You can setup hadoop to work on any user, but it is preferred that you setup a separate user for running hadoop.
sudo addgroup hadoop
sudo adduser -g hadoop hadoop - untar hadoop.tar.gz file in the user "hadoop's" home directory
[hadoop@linuxbox ~]$ tar -xvzf hadoop.tar.gz - check version of java - it should be atleast java 1.5 - preferred java 1.6
$ java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode) - Hadoop requires to ssh to the local server. So you would need to creat keys on local machine so that the ssh does not require password.
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
fb:7a:cf:c5:c0:ec:30:a7:f9:eb:f0:a4:8b:da:6f:88 hadoop@linuxbox
now copy the public key to the authorized_keys file, so that ssh should not require passwords
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Now check
$ ssh localhost
Last login: Sat Oct 18 18:30:57 2008 from localhost
$ - Change environment parameters in hadoop-env.sh
export JAVA_HOME=/path/to/jdk_home_dir - Change configuration parameters in hadoop-site.xml.
<configuration>
in core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
in mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
We will skip running actual map-reduce tasks on a single node setup and go ahead with a multi-node setup. Once we have 2 machines up and running, we will run some example map-reduce tasks on those nodes. So, lets proceed with multi-node setup.
For multi node setup, you should have 2 machines up and running and both having hadoop - single node setup on them. We will refer the machines as master and slave, And assume that hadoop has been installed under /home/hadoop directory in both the nodes.
To shut down the hadoop cluste run the following on master
[master ~/hadoop]$ ./bin/stop-mapred.sh # to stop mapreduce daemons
[master ~/hadoop]$ ./bin/stop-dfs.sh # to stop the hdfs daemons
Now, lets populate some files on the hdfs and see if we can run some programs
Get the following files on your local filesystem in some test directory on master
[master ~/test]$ wget http://www.gutenberg.org/files/20417/20417-8.txt
[master ~/test]$ wget http://www.gutenberg.org/dirs/etext04/7ldvc10.txt
[master ~/test]$ wget http://www.gutenberg.org/files/4300/4300-8.txt
[master ~/test]$ wget http://www.gutenberg.org/dirs/etext99/advsh12.txt
Populate the files in the hdfs file system
[master ~/hadoop]$ ./bin/hadoop dfs -copyFromLocal ../test/ test
Check the files on the hdfs file system
[master ~/hadoop]$ ./bin/hadoop dfs -ls
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2008-10-20 12:37 /user/hadoop/test
[master ~/hadoop]$ ./bin/hadoop dfs -ls test
Found 4 items
-rw-r--r-- 2 hadoop supergroup 674425 2008-10-20 12:37 /user/hadoop/test/20417-8.txt
-rw-r--r-- 2 hadoop supergroup 1573048 2008-10-20 12:37 /user/hadoop/test/4300-8.txt
-rw-r--r-- 2 hadoop supergroup 1423808 2008-10-20 12:37 /user/hadoop/test/7ldvc10.txt
-rw-r--r-- 2 hadoop supergroup 590093 2008-10-20 12:37 /user/hadoop/test/advsh12.txt
Now lets run some test programs. Lets run the wordcount example and collect the output in the test-op directory.
[master ~/hadoop]$ ./bin/hadoop jar hadoop-0.18.1-examples.jar wordcount test test-op
08/10/20 12:49:45 INFO mapred.FileInputFormat: Total input paths to process : 4
08/10/20 12:49:46 INFO mapred.FileInputFormat: Total input paths to process : 4
08/10/20 12:49:46 INFO mapred.JobClient: Running job: job_200810201146_0003
08/10/20 12:49:47 INFO mapred.JobClient: map 0% reduce 0%
08/10/20 12:49:52 INFO mapred.JobClient: map 50% reduce 0%
08/10/20 12:49:56 INFO mapred.JobClient: map 100% reduce 0%
08/10/20 12:50:02 INFO mapred.JobClient: map 100% reduce 16%
08/10/20 12:50:05 INFO mapred.JobClient: Job complete: job_200810201146_0003
08/10/20 12:50:05 INFO mapred.JobClient: Counters: 16
08/10/20 12:50:05 INFO mapred.JobClient: File Systems
08/10/20 12:50:05 INFO mapred.JobClient: HDFS bytes read=4261374
08/10/20 12:50:05 INFO mapred.JobClient: HDFS bytes written=949192
08/10/20 12:50:05 INFO mapred.JobClient: Local bytes read=2044286
08/10/20 12:50:05 INFO mapred.JobClient: Local bytes written=3757882
08/10/20 12:50:05 INFO mapred.JobClient: Job Counters
08/10/20 12:50:05 INFO mapred.JobClient: Launched reduce tasks=1
08/10/20 12:50:05 INFO mapred.JobClient: Launched map tasks=4
08/10/20 12:50:05 INFO mapred.JobClient: Data-local map tasks=4
08/10/20 12:50:05 INFO mapred.JobClient: Map-Reduce Framework
08/10/20 12:50:05 INFO mapred.JobClient: Reduce input groups=88307
08/10/20 12:50:05 INFO mapred.JobClient: Combine output records=205890
08/10/20 12:50:05 INFO mapred.JobClient: Map input records=90949
08/10/20 12:50:05 INFO mapred.JobClient: Reduce output records=88307
08/10/20 12:50:05 INFO mapred.JobClient: Map output bytes=7077676
08/10/20 12:50:05 INFO mapred.JobClient: Map input bytes=4261374
08/10/20 12:50:05 INFO mapred.JobClient: Combine input records=853602
08/10/20 12:50:05 INFO mapred.JobClient: Map output records=736019
08/10/20 12:50:05 INFO mapred.JobClient: Reduce input records=88307
Now, lets check the output.
[master ~/hadoop]$ ./bin/hadoop dfs -ls test-op
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2008-10-20 12:45 /user/hadoop/test-op/_logs
-rw-r--r-- 2 hadoop supergroup 949192 2008-10-20 12:46 /user/hadoop/test-op/part-00000
[master ~/hadoop]$ ./bin/hadoop dfs -copyToLocal test-op/part-00000 test-op-part-00000
[master ~/hadoop]$ head test-op-part-00000
"'A 1
"'About 1
"'Absolute 1
"'Ah!' 2
"'Ah, 2
"'Ample.' 1
"'And 10
"'Arthur!' 1
"'As 1
"'At 1
That's it... We have a live setup of hadoop running on two machines...
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
in mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
- Format the name node
$ cd /home/hadoop
$ ./bin/hadoop namenode -format
Check the output for errors. - Start single node cluster
$ <HADOOP_INSTALL>/bin/start-all.sh
This should start the namenode, datanode, jobtracker and tasktracker - all on one machine. - Check whether the nodes are up and running. The output should be approximately like
$ jps
28982 JobTracker
28737 DataNode
28615 NameNode
30570 Jps
29109 TaskTracker
28870 SecondaryNameNode - In case of any error, Please check the log files in the <HADOOP_INSTALL_DIR>/logs directory.
- To stop the node, run
$ <HADOOP_INSTALL_DIR>/bin/stop-all.sh
We will skip running actual map-reduce tasks on a single node setup and go ahead with a multi-node setup. Once we have 2 machines up and running, we will run some example map-reduce tasks on those nodes. So, lets proceed with multi-node setup.
For multi node setup, you should have 2 machines up and running and both having hadoop - single node setup on them. We will refer the machines as master and slave, And assume that hadoop has been installed under /home/hadoop directory in both the nodes.
- Firstly stop the single node hadoop running on them.
$ <HADOOP_INSTALL_DIR>/bin/stop-all.sh - Edit /etc/hosts file on both the servers to setup master and slave names. Eg:
aaa.bbb.ccc.ddd master
www.xxx.yyy.zzz slave - Now, the master should be able to ssh to the slave server without any password, so copy the public key of master to that of slave.
master]$ ssh-keygen -t rsa
master ~/.ssh]$ scp id_rsa.pub hadoop@slave:.ssh/
slave ~/.ssh]$ cat id_rsa.pub >> authorized_keys
Test the ssh setup
master]$ ssh master
master]$ ssh slave
slave ]$ ssh slave - Change the <HADOOP_INSTALL_DIR>/conf/masters & <HADOOP_INSTALL_DIR>/conf/slaves file to add the master & slave hosts there on the master server. The files should look like this.
master ~/hadoop/conf]$ cat masters
master
The slaves file contains the hosts(one per line) where hadoop slave daemons (data nodes and task trackers) would run. In our case we are running the datanode & tasktracker on both machines. In addition the master server would also run the master related services (namenode). Both master and slave would store data.
master ~/hadoop/conf]$ cat slaves
master
slave - Now change the configuration (<HADOOP_INSTALL_DIR>/conf/hadoop-site.xml) on all machines (master & slave). Set/change the following variables.
Specify the host and port of the name node(master server).
fs.default.name = hdfs://master:54310
Specify the host and port of the job tracker (map reduce master).
mapred.job.tracker = master:54311
Specify the number of machines a single file should be replicated to before it becomes available. It should be equal to the number of slave nodes. In our case it is 2 (master & slave - both act as slaves as well).
dfs.replication = 2 - You need to format the namenode recreate the datanode. Do the following
master ~/hadoop/hadoop-hadoop]$ rm -rf dfs mapred
slave ~/hadoop/hadoop-hadoop]$ rm -rf dfs mapred
Recreate/reformat the name node
master ~/hadoop] $ ./bin/hadoop namenode -format - Start the cluster.
[master ~/hadoop]$ ./bin/start-dfs.sh
starting namenode, logging to /home/hadoop/bin/../logs/hadoop-hadoop-namenode-master.out
master: starting datanode, logging to /home/hadoop/bin/../logs/hadoop-hadoop-datanode-master.out
slave: starting datanode, logging to /home/hadoop/bin/../logs/hadoop-hadoop-datanode-slave.out
master: starting secondarynamenode, logging to /home/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-master.out
Check the processes running on the master node
[master ~/hadoop]$ jps
5249 SecondaryNameNode
5319 Jps
5117 DataNode
4995 NameNode
Check the processes running on slave node
[slave ~/hadoop]$ jps
22256 Jps
22203 DataNode
Check the logs on the slave for errors. <HADOOP_INSTALL_DIR>/logs/hadoop-hadoop-datanode-slave.log - Now start the mapreduce daemons:
[master ~/hadoop]$ ./bin/start-mapred.sh
starting jobtracker, logging to /home/hadoop/bin/../logs/hadoop-hadoop-jobtracker-master.out
slave: starting tasktracker, logging to /home/hadoop/bin/../logs/hadoop-hadoop-tasktracker-slave.out
master: starting tasktracker, logging to /home/hadoop/bin/../logs/hadoop-hadoop-tasktracker-master.out
Check the processes on master
[master ~/hadoop]$ jps
5249 SecondaryNameNode
5117 DataNode
5725 TaskTracker
5598 JobTracker
5853 Jps
4995 NameNode
And the processes on the slave
[slave ~/hadoop]$ jps
22735 TaskTracker
22856 Jps
22413 DataNode
To shut down the hadoop cluste run the following on master
[master ~/hadoop]$ ./bin/stop-mapred.sh # to stop mapreduce daemons
[master ~/hadoop]$ ./bin/stop-dfs.sh # to stop the hdfs daemons
Now, lets populate some files on the hdfs and see if we can run some programs
Get the following files on your local filesystem in some test directory on master
[master ~/test]$ wget http://www.gutenberg.org/files/20417/20417-8.txt
[master ~/test]$ wget http://www.gutenberg.org/dirs/etext04/7ldvc10.txt
[master ~/test]$ wget http://www.gutenberg.org/files/4300/4300-8.txt
[master ~/test]$ wget http://www.gutenberg.org/dirs/etext99/advsh12.txt
Populate the files in the hdfs file system
[master ~/hadoop]$ ./bin/hadoop dfs -copyFromLocal ../test/ test
Check the files on the hdfs file system
[master ~/hadoop]$ ./bin/hadoop dfs -ls
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2008-10-20 12:37 /user/hadoop/test
[master ~/hadoop]$ ./bin/hadoop dfs -ls test
Found 4 items
-rw-r--r-- 2 hadoop supergroup 674425 2008-10-20 12:37 /user/hadoop/test/20417-8.txt
-rw-r--r-- 2 hadoop supergroup 1573048 2008-10-20 12:37 /user/hadoop/test/4300-8.txt
-rw-r--r-- 2 hadoop supergroup 1423808 2008-10-20 12:37 /user/hadoop/test/7ldvc10.txt
-rw-r--r-- 2 hadoop supergroup 590093 2008-10-20 12:37 /user/hadoop/test/advsh12.txt
Now lets run some test programs. Lets run the wordcount example and collect the output in the test-op directory.
[master ~/hadoop]$ ./bin/hadoop jar hadoop-0.18.1-examples.jar wordcount test test-op
08/10/20 12:49:45 INFO mapred.FileInputFormat: Total input paths to process : 4
08/10/20 12:49:46 INFO mapred.FileInputFormat: Total input paths to process : 4
08/10/20 12:49:46 INFO mapred.JobClient: Running job: job_200810201146_0003
08/10/20 12:49:47 INFO mapred.JobClient: map 0% reduce 0%
08/10/20 12:49:52 INFO mapred.JobClient: map 50% reduce 0%
08/10/20 12:49:56 INFO mapred.JobClient: map 100% reduce 0%
08/10/20 12:50:02 INFO mapred.JobClient: map 100% reduce 16%
08/10/20 12:50:05 INFO mapred.JobClient: Job complete: job_200810201146_0003
08/10/20 12:50:05 INFO mapred.JobClient: Counters: 16
08/10/20 12:50:05 INFO mapred.JobClient: File Systems
08/10/20 12:50:05 INFO mapred.JobClient: HDFS bytes read=4261374
08/10/20 12:50:05 INFO mapred.JobClient: HDFS bytes written=949192
08/10/20 12:50:05 INFO mapred.JobClient: Local bytes read=2044286
08/10/20 12:50:05 INFO mapred.JobClient: Local bytes written=3757882
08/10/20 12:50:05 INFO mapred.JobClient: Job Counters
08/10/20 12:50:05 INFO mapred.JobClient: Launched reduce tasks=1
08/10/20 12:50:05 INFO mapred.JobClient: Launched map tasks=4
08/10/20 12:50:05 INFO mapred.JobClient: Data-local map tasks=4
08/10/20 12:50:05 INFO mapred.JobClient: Map-Reduce Framework
08/10/20 12:50:05 INFO mapred.JobClient: Reduce input groups=88307
08/10/20 12:50:05 INFO mapred.JobClient: Combine output records=205890
08/10/20 12:50:05 INFO mapred.JobClient: Map input records=90949
08/10/20 12:50:05 INFO mapred.JobClient: Reduce output records=88307
08/10/20 12:50:05 INFO mapred.JobClient: Map output bytes=7077676
08/10/20 12:50:05 INFO mapred.JobClient: Map input bytes=4261374
08/10/20 12:50:05 INFO mapred.JobClient: Combine input records=853602
08/10/20 12:50:05 INFO mapred.JobClient: Map output records=736019
08/10/20 12:50:05 INFO mapred.JobClient: Reduce input records=88307
Now, lets check the output.
[master ~/hadoop]$ ./bin/hadoop dfs -ls test-op
Found 2 items
drwxr-xr-x - hadoop supergroup 0 2008-10-20 12:45 /user/hadoop/test-op/_logs
-rw-r--r-- 2 hadoop supergroup 949192 2008-10-20 12:46 /user/hadoop/test-op/part-00000
[master ~/hadoop]$ ./bin/hadoop dfs -copyToLocal test-op/part-00000 test-op-part-00000
[master ~/hadoop]$ head test-op-part-00000
"'A 1
"'About 1
"'Absolute 1
"'Ah!' 2
"'Ah, 2
"'Ample.' 1
"'And 10
"'Arthur!' 1
"'As 1
"'At 1
That's it... We have a live setup of hadoop running on two machines...
Saturday, October 11, 2008
install latest vlc on ubuntu 8.04
So, you have got ubuntu and you are still playing movies on 0.8.6 something version. No matter how many times you do an "apt-get update", vlc does not upgrade. The way to do this is:
a) sudo vim /etc/apt/sources.list
b) Add line "deb http://ppa.launchpad.net/c-korn/ubuntu hardy main" and save the file
c) sudo apt-get update
d) sudo apt-get install vlc
Now when you type in vlc, you will see the latest vlc player popping up...
a) sudo vim /etc/apt/sources.list
b) Add line "deb http://ppa.launchpad.net/c-korn/ubuntu hardy main" and save the file
c) sudo apt-get update
d) sudo apt-get install vlc
Now when you type in vlc, you will see the latest vlc player popping up...
how law and order works in india...
It had been a week since we had got vegetables from the market. For the past 3-4 days we had been surviving only on varieties made out of potatoes and onions. So, on a fine monday evening, while we were driving back to home from office, we thought that we should get down at the roadside vegetable market and get some veggies. So, we stopped on the opposite side of the road from where the market was, got down and went to get the veggies. There were lots of cars parked over there and ofcourse it is a very busy road with cars and other vehicles moving up and down. Also tons of rickshaw-walas were waiting not 100 meters from where i had my car parked.
We are really quick shoppers, so in 15 minutes we had got supplies for almost a week and we decided that we should move back. When i came back, i saw that the window of the right hand back door was not there. Well, it was there, but it was in pieces and my precious laptop bag with all its contents was missing. It took some time for the fact to sink in that my bag was missing. I looked left and right for the f*** who had stolen my bag, but of-course, no one could be found. And then i looked at people nearby. I saw a man sitting on a chair 10 meters from my car. I ran to him and asked if he has seen anything. Ofcourse, he had seen nothing. The rickshaw-walas were totally ignorant and emphatetic with my situation. "ka jamana aa gaya hai, seesha tod kar bagwa le kar chala gaya". That is how they express their grief.
In india, when things are stolen from you, you have to give up all hope of getting it back. People generally turn blind and deaf when they feel that something wrong is happening in their surrounding. The reason behind this might be the fact that the probability of getting caught and facing a sentence is very low (maybe 0.1 %). People generally try to avoid reporting crimes, because the police are totally un-cooperative. The police try "NOT" to find lost things. I cant figure out if it is their laziness or lack of IQ. It might be both. In foreign countries, say U.S., you dial 911 and within 5-10 minutes, you have cops at your door - trying to help you out. I have not tried dialing 100 here, but i dont think that the police would be at the place in less than an hour.
Well, lets move ahead with the story. So, i called up one of my friends and told him what had happened. He told me that he would be home in an hour and then we can go and maybe report the crime - that is if it is required to claim the insurance. The main things I had lost were my official laptop, possession letter (I had specially taken it out on that day to get it photocopied), RC of my bike, bank and credit-card statements and an almost new cheque book. I went home and made a complete list of things i have lost and passwords that need changing. ( I had stored some passwords on my laptop).
An hour later, me and my friend wrote down an application in english and went to the nearby police "chowki" to get the report registered. The first reaction of the policeman sitting there was that "why was the application in english?", so we wrote it again in Hindi. Imagine the policemen unable to read english. What would happen if a foreign tourist gets robbed? Would he be able to even make these guys understand what has happened? When we handed over the freshly written application to him, he simply put it on the table and went out to get his superior officer. The superior officer read the application and asked us to take him to the place where the incident has happened. I had used my brain a little and had safely put away my car and taken my bike. Mainly because i was not sure if they would ask me to let the car be with them or they would ask more money looking at a long car and thinking that i am a "rich" man.
So, we rode after them on the motorcycle and went to the vegetable market. I showed him the place where i had parked the car from a distance - i dont know why he did not go to the place. And told him that there were rickshaws and a man sitting nearby, he was quick to ask whether i suspect the man to be the thief. How could i judge? Should he not question the person and check out whether he was right or wrong? Anyways, next he started blaming me for parking my car on the road where there was no parking. But, there is no parking space nearby - i thought. And we came back to his "chowki". He kept the application and told us to check on the status next morning. We asked him if he would give us an FIR. But the reply was "No" - "mai FIR nahi likhta".
I knew this would happen and it would be difficult to get an FIR our of these guys without giving them some "donation" in return. But i had expected them to ask upfront for a "donation". But no such thing happened, so we waited. We waited for almost 15 minutes, but nothing happened. Then the "daroga" got angry because we were waiting for his response and said "Ab laptop le kar hi jaogay kya? Dekh lo kahi yehi pada ho to le jana.". We did not know what to reply, so we simply went home.
Next day, i went to the police station with 2 of my friends and again repeated the complete story to atleast 2 police officers. One of them told us to get our car to show the damage. So, we went and got the car. He then sent some junior havaldar to check out whether the window was broken (that is whether we were telling the truth or not). Then we were asked to consult the SO (head in charge of the police station). Again we repeated the story to him. And showed him the car from a distance. So, he told his junior officers to accept the "application" [Still no FIR]. The junior officer simply accepted the application and stamped it and drew a vertical line (which i believe was his signature). He did not read the contents, neither did he check what language it was written in.
After accepting the application, when we asked for a complaint no, he stopped looking at us and started shifting papers from one box to the same box. I think, he was trying to ignore us - i dont know why? So, we again went to the SO and told him that we wanted some complaint no. And he redirected us to another officer, to whom we again repeated the whole story and showed him the application. He simply said "yeah to angreji mein hai. Isko received kisne kia?" (this is in english, how did this get accepted?) - how was i supposed to know the answer to this question? He took us back to the previous junior officer who had accepted and asked him why did he accept an application in english?. Well, the application was returned back to us and we were asked to rewrite it in hindi and submit it again.
So, we wrote it in hindi and submitted it again. This time the junior officer read it and then stamped it. It was during that time that we came to know that the junior officer was an 8th pass and did not know anything about english. If this is the type of education that a policeman has, then what can we expect out of them. We asked almost everyone about when can we get an FIR or a complaint no, and the response was "kal" (tomorrow). Someone even said that we might get in 2-3 hours, if we are ready to wait.
When i came to the office that day, lots of people came to me and shared their experiences when they had to get an FIR. Some of them had spent months to get an FIR. One very sad case had spent 200/- for the FIR and even after that, there was some mistake in the writing and so he had not yet been able to claim his insurance.
For the next 2 days, i just went and asked whether the FIR was ready and the general response was "kal" - mainly due to work pressure. Well, if the police are so busy writing FIR's who would do the investigation and catch the criminals. I think, they have a tough job to do - trying to write down so many FIR's instead of catching the criminals and reducing the crime rate. Everybody advised me to pay them to get the FIR. But when i offered them, they would say that it is unnecessary and still make me come the next day.
Finally on the 3rd day, dad was here, so he went with me and talked them into writing the FIR. We were again sent to the SO to whom we again repeated the complete story and reminded him about the talk we have had earlier. He again read the application that we had submitted and asked us to get a photocopy of the bill of the laptop. We rushed to the office and got a photocopy of the bill. This time the SO was generous and told us to get the FIR written. When we again went to the junior officer, he asked us to wait and then confirmed from the SO whether he should write the FIR. Again we were told to come after 2 hours. This time dad made an indication of the offer and i think the junior officer got it.
We were again asked to write a fresh application and change the dates accordingly. After some pestering the junior officer finally started writing the FIR. Finally after around 30 minutes the FIR was ready and we got a copy. After i came out, i asked dad whether he had given them the "donation" - because i did not see him giving it to them. Then dad told me that it was given to them when he shook hands with them for the final "thankyou".
The point here is that after being a victim to a crime, you have to be after the police to prove that you are a victim. Forget the option of getting back your stolen stuff. You have to give them some money to get the complaint registered. It is like "Please sir, (beg with folded hands - a 100/- rs note between the hands), i have suffered a great loss, please write my complaint". How can you expect justice to come out of these guys. I still remember the detective serials that I used to watch in my childhood. The actual scenario is worse than that. Policemen not only overlook valuable clues but dont want to find the criminal. If your car is picked up and left in the police station for a week, all you would get back would be the outer body and the seats. The steering, engine etc all would go missing. And if you enquire about it, all they would say is that it was brought here like that only.
The SO had his own style (tashan). He would keep on chewing paan, etc and keep on spitting out the reddish stuff. Specially for him there was a huge bucket to his left side, so that he could keep on spitting until the bucket is full and then it would be taken away.
Welcome to India...
We are really quick shoppers, so in 15 minutes we had got supplies for almost a week and we decided that we should move back. When i came back, i saw that the window of the right hand back door was not there. Well, it was there, but it was in pieces and my precious laptop bag with all its contents was missing. It took some time for the fact to sink in that my bag was missing. I looked left and right for the f*** who had stolen my bag, but of-course, no one could be found. And then i looked at people nearby. I saw a man sitting on a chair 10 meters from my car. I ran to him and asked if he has seen anything. Ofcourse, he had seen nothing. The rickshaw-walas were totally ignorant and emphatetic with my situation. "ka jamana aa gaya hai, seesha tod kar bagwa le kar chala gaya". That is how they express their grief.
In india, when things are stolen from you, you have to give up all hope of getting it back. People generally turn blind and deaf when they feel that something wrong is happening in their surrounding. The reason behind this might be the fact that the probability of getting caught and facing a sentence is very low (maybe 0.1 %). People generally try to avoid reporting crimes, because the police are totally un-cooperative. The police try "NOT" to find lost things. I cant figure out if it is their laziness or lack of IQ. It might be both. In foreign countries, say U.S., you dial 911 and within 5-10 minutes, you have cops at your door - trying to help you out. I have not tried dialing 100 here, but i dont think that the police would be at the place in less than an hour.
Well, lets move ahead with the story. So, i called up one of my friends and told him what had happened. He told me that he would be home in an hour and then we can go and maybe report the crime - that is if it is required to claim the insurance. The main things I had lost were my official laptop, possession letter (I had specially taken it out on that day to get it photocopied), RC of my bike, bank and credit-card statements and an almost new cheque book. I went home and made a complete list of things i have lost and passwords that need changing. ( I had stored some passwords on my laptop).
An hour later, me and my friend wrote down an application in english and went to the nearby police "chowki" to get the report registered. The first reaction of the policeman sitting there was that "why was the application in english?", so we wrote it again in Hindi. Imagine the policemen unable to read english. What would happen if a foreign tourist gets robbed? Would he be able to even make these guys understand what has happened? When we handed over the freshly written application to him, he simply put it on the table and went out to get his superior officer. The superior officer read the application and asked us to take him to the place where the incident has happened. I had used my brain a little and had safely put away my car and taken my bike. Mainly because i was not sure if they would ask me to let the car be with them or they would ask more money looking at a long car and thinking that i am a "rich" man.
So, we rode after them on the motorcycle and went to the vegetable market. I showed him the place where i had parked the car from a distance - i dont know why he did not go to the place. And told him that there were rickshaws and a man sitting nearby, he was quick to ask whether i suspect the man to be the thief. How could i judge? Should he not question the person and check out whether he was right or wrong? Anyways, next he started blaming me for parking my car on the road where there was no parking. But, there is no parking space nearby - i thought. And we came back to his "chowki". He kept the application and told us to check on the status next morning. We asked him if he would give us an FIR. But the reply was "No" - "mai FIR nahi likhta".
I knew this would happen and it would be difficult to get an FIR our of these guys without giving them some "donation" in return. But i had expected them to ask upfront for a "donation". But no such thing happened, so we waited. We waited for almost 15 minutes, but nothing happened. Then the "daroga" got angry because we were waiting for his response and said "Ab laptop le kar hi jaogay kya? Dekh lo kahi yehi pada ho to le jana.". We did not know what to reply, so we simply went home.
Next day, i went to the police station with 2 of my friends and again repeated the complete story to atleast 2 police officers. One of them told us to get our car to show the damage. So, we went and got the car. He then sent some junior havaldar to check out whether the window was broken (that is whether we were telling the truth or not). Then we were asked to consult the SO (head in charge of the police station). Again we repeated the story to him. And showed him the car from a distance. So, he told his junior officers to accept the "application" [Still no FIR]. The junior officer simply accepted the application and stamped it and drew a vertical line (which i believe was his signature). He did not read the contents, neither did he check what language it was written in.
After accepting the application, when we asked for a complaint no, he stopped looking at us and started shifting papers from one box to the same box. I think, he was trying to ignore us - i dont know why? So, we again went to the SO and told him that we wanted some complaint no. And he redirected us to another officer, to whom we again repeated the whole story and showed him the application. He simply said "yeah to angreji mein hai. Isko received kisne kia?" (this is in english, how did this get accepted?) - how was i supposed to know the answer to this question? He took us back to the previous junior officer who had accepted and asked him why did he accept an application in english?. Well, the application was returned back to us and we were asked to rewrite it in hindi and submit it again.
So, we wrote it in hindi and submitted it again. This time the junior officer read it and then stamped it. It was during that time that we came to know that the junior officer was an 8th pass and did not know anything about english. If this is the type of education that a policeman has, then what can we expect out of them. We asked almost everyone about when can we get an FIR or a complaint no, and the response was "kal" (tomorrow). Someone even said that we might get in 2-3 hours, if we are ready to wait.
When i came to the office that day, lots of people came to me and shared their experiences when they had to get an FIR. Some of them had spent months to get an FIR. One very sad case had spent 200/- for the FIR and even after that, there was some mistake in the writing and so he had not yet been able to claim his insurance.
For the next 2 days, i just went and asked whether the FIR was ready and the general response was "kal" - mainly due to work pressure. Well, if the police are so busy writing FIR's who would do the investigation and catch the criminals. I think, they have a tough job to do - trying to write down so many FIR's instead of catching the criminals and reducing the crime rate. Everybody advised me to pay them to get the FIR. But when i offered them, they would say that it is unnecessary and still make me come the next day.
Finally on the 3rd day, dad was here, so he went with me and talked them into writing the FIR. We were again sent to the SO to whom we again repeated the complete story and reminded him about the talk we have had earlier. He again read the application that we had submitted and asked us to get a photocopy of the bill of the laptop. We rushed to the office and got a photocopy of the bill. This time the SO was generous and told us to get the FIR written. When we again went to the junior officer, he asked us to wait and then confirmed from the SO whether he should write the FIR. Again we were told to come after 2 hours. This time dad made an indication of the offer and i think the junior officer got it.
We were again asked to write a fresh application and change the dates accordingly. After some pestering the junior officer finally started writing the FIR. Finally after around 30 minutes the FIR was ready and we got a copy. After i came out, i asked dad whether he had given them the "donation" - because i did not see him giving it to them. Then dad told me that it was given to them when he shook hands with them for the final "thankyou".
The point here is that after being a victim to a crime, you have to be after the police to prove that you are a victim. Forget the option of getting back your stolen stuff. You have to give them some money to get the complaint registered. It is like "Please sir, (beg with folded hands - a 100/- rs note between the hands), i have suffered a great loss, please write my complaint". How can you expect justice to come out of these guys. I still remember the detective serials that I used to watch in my childhood. The actual scenario is worse than that. Policemen not only overlook valuable clues but dont want to find the criminal. If your car is picked up and left in the police station for a week, all you would get back would be the outer body and the seats. The steering, engine etc all would go missing. And if you enquire about it, all they would say is that it was brought here like that only.
The SO had his own style (tashan). He would keep on chewing paan, etc and keep on spitting out the reddish stuff. Specially for him there was a huge bucket to his left side, so that he could keep on spitting until the bucket is full and then it would be taken away.
Welcome to India...
Wednesday, October 01, 2008
road rage
I have driven for hours in delhi. I have driven from noida to gurgaon in very heavy traffic which took me 3 hours to reach. Sometimes, it really becomes troublesome when you have a nature's call and you cannot do anything about it. At most of the places where jams are frequent, you could see vendors selling water and other stuff to eat. But there are no public toilets nearby.
Now-a-days, all cycle walas and rickshaw walas think that they are shahrukh khan. They ride in the middle of the road and try to over take your car, cutting in front of you - as if they are invincible. People on bike also believe that they are GOD. They think, they can jump red lights without getting into any accidents and ride in the middle of the road at a constant speed of 30 kmph - turning a deaf ear to your honking and not allowing you to overtake. And the best part is when you see circus like acts on the road with 3 people on a cycle or 3 people on a small bike. I sometimes feel mercy for the overloaded vehicles. And even think whether the people who had designed the engine of the bike had taken such situations into consideration.
This attitude does lead to some frustration. After driving for an hour to cover a 20 minute distance, trying to protect these people inspite of their own carelessness, you tend to get exhausted. And you also have to protect yourself from the buses which keep on zooming from
left and right both in wrong direction and right direction. What amazes me about these busses is how they drive by you with only 2 inches to spare between your vehicle and the bus, at full speed and still it misses you. Arent the bus drivers extremely talented.
And finally cows & dogs on the road. They would just stand there confused about where to go and what to do. The municipal corporation does not try to remove them from the road. Who cares, people would simply go around them.
And ofcourse, the most famous of all these are the tampos. Yup the green colored tampos which have CNG written on their back and still throw out tons and tons of black smoke. They are heavily overloaded and keep on ferrying people between points A & B. They dont go beyond 20 kmph and they always move in the middle of the road, so that all the traffic stay behind them. They always stop in the middle of the road and just at a crossing, so that they could create jams and make people acknowledge their importance by their honking.
When people encounter these situations, and are unable to arrive at a compromise, they simply blow up. Why should they care for people who do not care for themselves?
If i would have had a jeannie, i would have wished that it taught the people on the road to have more road sense.
Now-a-days, all cycle walas and rickshaw walas think that they are shahrukh khan. They ride in the middle of the road and try to over take your car, cutting in front of you - as if they are invincible. People on bike also believe that they are GOD. They think, they can jump red lights without getting into any accidents and ride in the middle of the road at a constant speed of 30 kmph - turning a deaf ear to your honking and not allowing you to overtake. And the best part is when you see circus like acts on the road with 3 people on a cycle or 3 people on a small bike. I sometimes feel mercy for the overloaded vehicles. And even think whether the people who had designed the engine of the bike had taken such situations into consideration.
This attitude does lead to some frustration. After driving for an hour to cover a 20 minute distance, trying to protect these people inspite of their own carelessness, you tend to get exhausted. And you also have to protect yourself from the buses which keep on zooming from
left and right both in wrong direction and right direction. What amazes me about these busses is how they drive by you with only 2 inches to spare between your vehicle and the bus, at full speed and still it misses you. Arent the bus drivers extremely talented.
And finally cows & dogs on the road. They would just stand there confused about where to go and what to do. The municipal corporation does not try to remove them from the road. Who cares, people would simply go around them.
And ofcourse, the most famous of all these are the tampos. Yup the green colored tampos which have CNG written on their back and still throw out tons and tons of black smoke. They are heavily overloaded and keep on ferrying people between points A & B. They dont go beyond 20 kmph and they always move in the middle of the road, so that all the traffic stay behind them. They always stop in the middle of the road and just at a crossing, so that they could create jams and make people acknowledge their importance by their honking.
When people encounter these situations, and are unable to arrive at a compromise, they simply blow up. Why should they care for people who do not care for themselves?
If i would have had a jeannie, i would have wished that it taught the people on the road to have more road sense.
Saturday, September 27, 2008
Broadband finally...
Yuppie... i got a broadband finally.
When i was a kid, i had this phone line on which i had a modem. I had to dial a particular no and then wait as the web pages loaded. Downloading songs and videos were a huge pain. I had to queue up the downloads and let my pc run overnight to download a few songs (a few mbs). So, when i grew up, i had this thought that i should be getting my own broadband connection and do tons and tons of downloads and uploads - maybe put my own server with a dedicated ip address & dns entry and run maybe some web services on it.
After some research/enquiry i came to know that out of all the broadbands, airtel was supposed to be the best. So i called up an airtel salu and asked him to put my broadband in place. He came, took some documents and promised that i would get my connection the next day. Next day - nothing happened. The day after next, i called him up and he told me that today the task would be done. The same day some wiring people came and wired up my whole place for phone and internet. The appartment that i am in, might be having some pre-arranged setup with airtel, cause all the internal wiring was done. All they had to do was put in connectors and connect them to the internal wires which was already in place.
So, all my rooms [3 bedrooms + 1 drawing room] now have 2 extra connectors - for phone and internet. And there are no extra wires to be seen. I did not opt for wifi - the main reason because it is sometimes difficult to configure wifi on linux [I had configured it on my laptop, but it stopped working some time back after ubuntu sent in some updates] - and i did not want my family members to enjoy wifi while i connect to the net using wires. (a bit selfish on me, but thats the way i am).
After the wiring, the waiting started - for the engineers to turn up, install the device and start the broadband connection. Everyday i used to call up the salu, but nothing happened. Then i asked him about the engineers' no and called up the engineer. Next day the guys turned up and it took them 3 hours to get the net up and running - they had to do some setting from their backend which took a lot of time due to lack of coordination.
Anyways, finally I have broadband in place and i have been downloading tons of softwares on it. Will start downloading music and videos as well. The speed is very good - at least better than what i used to get on reliance aircard.
When i was a kid, i had this phone line on which i had a modem. I had to dial a particular no and then wait as the web pages loaded. Downloading songs and videos were a huge pain. I had to queue up the downloads and let my pc run overnight to download a few songs (a few mbs). So, when i grew up, i had this thought that i should be getting my own broadband connection and do tons and tons of downloads and uploads - maybe put my own server with a dedicated ip address & dns entry and run maybe some web services on it.
After some research/enquiry i came to know that out of all the broadbands, airtel was supposed to be the best. So i called up an airtel salu and asked him to put my broadband in place. He came, took some documents and promised that i would get my connection the next day. Next day - nothing happened. The day after next, i called him up and he told me that today the task would be done. The same day some wiring people came and wired up my whole place for phone and internet. The appartment that i am in, might be having some pre-arranged setup with airtel, cause all the internal wiring was done. All they had to do was put in connectors and connect them to the internal wires which was already in place.
So, all my rooms [3 bedrooms + 1 drawing room] now have 2 extra connectors - for phone and internet. And there are no extra wires to be seen. I did not opt for wifi - the main reason because it is sometimes difficult to configure wifi on linux [I had configured it on my laptop, but it stopped working some time back after ubuntu sent in some updates] - and i did not want my family members to enjoy wifi while i connect to the net using wires. (a bit selfish on me, but thats the way i am).
After the wiring, the waiting started - for the engineers to turn up, install the device and start the broadband connection. Everyday i used to call up the salu, but nothing happened. Then i asked him about the engineers' no and called up the engineer. Next day the guys turned up and it took them 3 hours to get the net up and running - they had to do some setting from their backend which took a lot of time due to lack of coordination.
Anyways, finally I have broadband in place and i have been downloading tons of softwares on it. Will start downloading music and videos as well. The speed is very good - at least better than what i used to get on reliance aircard.
Wednesday, September 24, 2008
Softwareism
Chandrababuism
You have two cows in Vijayawada. You hook them to internet and milk them from Hyderabad .
Jayalalithaism
You have two cows. You teach them to cry,"Ammaaaaaaa..."
and fall at your feet.
Karunanidhiism
You have two cows. You give one to your son and the other to your nephew .
.
Gandhism
You have two cows. But you drink goat's milk.
Indiraism
You have two bulls. You adamantly consider them as cows.
Lalooism
You have two cows. You buy Rs. 900 Crore worth of cattlefeed for them.
Rajnikantism
You have two cows. You throw them into air and catch their milk in your mouth.
Sardarism
You have two cows. You paint them both to get colourful milk.
Softwarism: (Ultimate....)
Client has 2 cows and u need to milk them.
1 .. First prepare a document when to milk them (Project kick off)
2 .. Prepare a document how long you have to milk them (Project plan)
3 .. Then prepare how to milk them (Design)
4 .. Then prepare what other accessories are needed to milk them (Framework)
5 .. Then prepare a 2 dummy cows (sort of toy cows) and show to client the way in which u will milk them (UI Mockups & POC)
6 .. If client is not satisfied then redo from step 2
7 You actually start milking them and find that there are few problem with accessories. (Change framework)
8 .. Redo step 4
9 .. At last milk them and send it to onsite. (Coding over)
10. Make sure that cow milks properly ( Testing)
11. Onsite reports that it is not milking there.
12. You break your head and find that onsite is trying to milk from bulls
13. At last onsite milk them and send to client (Testing)
14. Client says the quality of milk is not good. (User Acceptance Test)
15. Offsite then slogs and improves the quality of milk
16. Now the client says that the quality is good but its milking at slow rate (performance issue)
17. Again you slog and send it with good performance.
18. Client is happy???
By this time both the COWs aged and cant milk.
(The software got old and get ready for next release
repeat from step 1) !!!!
You have two cows in Vijayawada. You hook them to internet and milk them from Hyderabad .
Jayalalithaism
You have two cows. You teach them to cry,"Ammaaaaaaa..."
and fall at your feet.
Karunanidhiism
You have two cows. You give one to your son and the other to your nephew .
.
Gandhism
You have two cows. But you drink goat's milk.
Indiraism
You have two bulls. You adamantly consider them as cows.
Lalooism
You have two cows. You buy Rs. 900 Crore worth of cattlefeed for them.
Rajnikantism
You have two cows. You throw them into air and catch their milk in your mouth.
Sardarism
You have two cows. You paint them both to get colourful milk.
Softwarism: (Ultimate....)
Client has 2 cows and u need to milk them.
1 .. First prepare a document when to milk them (Project kick off)
2 .. Prepare a document how long you have to milk them (Project plan)
3 .. Then prepare how to milk them (Design)
4 .. Then prepare what other accessories are needed to milk them (Framework)
5 .. Then prepare a 2 dummy cows (sort of toy cows) and show to client the way in which u will milk them (UI Mockups & POC)
6 .. If client is not satisfied then redo from step 2
7 You actually start milking them and find that there are few problem with accessories. (Change framework)
8 .. Redo step 4
9 .. At last milk them and send it to onsite. (Coding over)
10. Make sure that cow milks properly ( Testing)
11. Onsite reports that it is not milking there.
12. You break your head and find that onsite is trying to milk from bulls
13. At last onsite milk them and send to client (Testing)
14. Client says the quality of milk is not good. (User Acceptance Test)
15. Offsite then slogs and improves the quality of milk
16. Now the client says that the quality is good but its milking at slow rate (performance issue)
17. Again you slog and send it with good performance.
18. Client is happy???
By this time both the COWs aged and cant milk.
(The software got old and get ready for next release
repeat from step 1) !!!!
Monday, August 25, 2008
Thread Executors in java
With Java 1.4, when we used to write threaded applications, we did not have much options in limiting and reusing the same thread for the available tasks. I had written applications earlier where each process that needed to be executed freely (or a process which can be executed on a thread) used to create its own thread. And once a thread was created and used, it was simply discarded. New threads were created whenever needed.
With java 1.5, there is an executor class which allows you to create controlled thread pools and execute the list of tasks on the same number of threads. If the number of tasks exceeds the number of threads then these extra tasks just wait for a thread to become available and then start their execution.
A simple threaded server using java 1.4 :
1 import java.io.*;
2 import java.net.*;
3 import java.util.*;
4
5 public class testPool implements Runnable
6 {
7 private int port;
8 private int active, total;
9
10 public testPool(int port)
11 {
12 this.port = port;
13 System.out.println("Thread pool constructor - creating thread");
14 new Thread(this).start();
15 }
16
17 public void run()
18 {
19 try
20 {
21 System.out.println("Thread pool - new thread created");
22 ServerSocket ss = new ServerSocket(port);
23 while(true)
24 {
25 Socket s = ss.accept();
26 total++;
27 new Handler(s);
28 }
29 }catch(Exception ex)
30 {
31 ex.printStackTrace();
32 }
33 }
34
35 public class Handler implements Runnable
36 {
37 private Socket socket;
38 public Handler(Socket s)
39 {
40 this.socket = s;
41 System.out.println("Handler constructor - creating thread");
42 new Thread(this).start();
43 }
44 public void run()
45 {
46 System.out.println(Thread.currentThread().getName()+" - Handler - new thread created");
47 active++;
48 boolean loop = true;
49 try
50 {
51 BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
52 DataOutputStream out = new DataOutputStream(socket.getOutputStream());
53 while(loop)
54 {
55 String cmd = in.readLine();
56 if(cmd.equals("QUIT"))
57 {
58 loop = false;
59 socket.close();
60 }
61 else if(cmd.equals("INFO"))
62 {
63 System.out.println("Active Connections : "+active);
64 System.out.println("Total Connections : "+total);
65 }
66 else
67 {
68 System.out.println("Command = "+cmd);
69 }
70 }
71 }catch(Exception ex)
72 {
73 ex.printStackTrace();
74 }
75 active--;
76 }
77 }
78
79 public static void main(String[] args)
80 {
81 System.out.println("Starting Daemon");
82 new testPool(Integer.parseInt(args[0]));
83 }
84 }
Compile and run the program
jayant@jayantbox:~/myprogs/java$ java -cp . testPool 5000
Starting Daemon
Thread pool constructor - creating thread
Thread pool - new thread created
Now, as you keep on connecting to localhost 5000 port, you can see that new threads are created. Even when the old connections are closed, the same thread is not used again but new threads are created.
so, if i do
telnet localhost 5000
from 3 consoles and issue the "INFO" command from 4th console, the output is
jayant@jayantbox:~/myprogs/java$ java -cp . testPool 5000
Starting Daemon
Thread pool constructor - creating thread
Thread pool - new thread created
Handler constructor - creating thread
Thread-1 - Handler - new thread created
Handler constructor - creating thread
Thread-2 - Handler - new thread created
Handler constructor - creating thread
Thread-3 - Handler - new thread created
Handler constructor - creating thread
Thread-4 - Handler - new thread created
Active Connections : 1
Total Connections : 4
4 threads are created. In this case, everytime you connect to the server, a new thread would be created.
Now, lets modify the code to work on java 1.5 using thread Executors...
Add the following code snippets at or after the following line numbers
4 import java.util.concurrent.*;
10 private int pool;
12 this.pool=2;
22 ExecutorService threadExecutor = Executors.newFixedThreadPool(pool);
27 threadExecutor.execute(new Handler(s));
And comment the following 2 lines
27 new Handler(s);
42 new Thread(this).start();
Not lets run the program again on port 5000 and try connecting to it. We have created a pool of 2 threads here. Observer that:
Process for testing
console 1: telnet localhost 5000
console 2: telnet localhost 5000
console 3: telnet localhost 5000
console 1: from 1
console 2: from 2
console 3: from 3
console 1: INFO
console 1: 1 quitting
console 1: QUIT
Output for the testing process
jjayant@jayantbox:~/myprogs/java$ java -cp . testPool 5000
Starting Daemon
Thread pool constructor - creating thread
Thread pool - new thread created
Handler constructor - creating thread
pool-1-thread-1 - Handler - new thread created
Handler constructor - creating thread
pool-1-thread-2 - Handler - new thread created
Handler constructor - creating thread
Command = from 1
Command = from 2
Active Connections : 2
Total Connections : 3
Command = 1 quitting
pool-1-thread-1 - Handler - new thread created
Command = from 3
As seen, processing of connection 3 starts only after connection 1 is closed by the client.
The benefits of using a thread executor instead of a normal thread are
a] Overhead of creating and destroying threads is avoided.
b] A queue of threads is created automatically when the no of tasks exceeds the no of threads in the pool. The tasks in queue are automatically executed when the threads become free.
c] A limit (maximum no of threads) could be imposed thus controlling the available resources for delivering better performance.
With java 1.5, there is an executor class which allows you to create controlled thread pools and execute the list of tasks on the same number of threads. If the number of tasks exceeds the number of threads then these extra tasks just wait for a thread to become available and then start their execution.
A simple threaded server using java 1.4 :
1 import java.io.*;
2 import java.net.*;
3 import java.util.*;
4
5 public class testPool implements Runnable
6 {
7 private int port;
8 private int active, total;
9
10 public testPool(int port)
11 {
12 this.port = port;
13 System.out.println("Thread pool constructor - creating thread");
14 new Thread(this).start();
15 }
16
17 public void run()
18 {
19 try
20 {
21 System.out.println("Thread pool - new thread created");
22 ServerSocket ss = new ServerSocket(port);
23 while(true)
24 {
25 Socket s = ss.accept();
26 total++;
27 new Handler(s);
28 }
29 }catch(Exception ex)
30 {
31 ex.printStackTrace();
32 }
33 }
34
35 public class Handler implements Runnable
36 {
37 private Socket socket;
38 public Handler(Socket s)
39 {
40 this.socket = s;
41 System.out.println("Handler constructor - creating thread");
42 new Thread(this).start();
43 }
44 public void run()
45 {
46 System.out.println(Thread.currentThread().getName()+" - Handler - new thread created");
47 active++;
48 boolean loop = true;
49 try
50 {
51 BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
52 DataOutputStream out = new DataOutputStream(socket.getOutputStream());
53 while(loop)
54 {
55 String cmd = in.readLine();
56 if(cmd.equals("QUIT"))
57 {
58 loop = false;
59 socket.close();
60 }
61 else if(cmd.equals("INFO"))
62 {
63 System.out.println("Active Connections : "+active);
64 System.out.println("Total Connections : "+total);
65 }
66 else
67 {
68 System.out.println("Command = "+cmd);
69 }
70 }
71 }catch(Exception ex)
72 {
73 ex.printStackTrace();
74 }
75 active--;
76 }
77 }
78
79 public static void main(String[] args)
80 {
81 System.out.println("Starting Daemon");
82 new testPool(Integer.parseInt(args[0]));
83 }
84 }
Compile and run the program
jayant@jayantbox:~/myprogs/java$ java -cp . testPool 5000
Starting Daemon
Thread pool constructor - creating thread
Thread pool - new thread created
Now, as you keep on connecting to localhost 5000 port, you can see that new threads are created. Even when the old connections are closed, the same thread is not used again but new threads are created.
so, if i do
telnet localhost 5000
from 3 consoles and issue the "INFO" command from 4th console, the output is
jayant@jayantbox:~/myprogs/java$ java -cp . testPool 5000
Starting Daemon
Thread pool constructor - creating thread
Thread pool - new thread created
Handler constructor - creating thread
Thread-1 - Handler - new thread created
Handler constructor - creating thread
Thread-2 - Handler - new thread created
Handler constructor - creating thread
Thread-3 - Handler - new thread created
Handler constructor - creating thread
Thread-4 - Handler - new thread created
Active Connections : 1
Total Connections : 4
4 threads are created. In this case, everytime you connect to the server, a new thread would be created.
Now, lets modify the code to work on java 1.5 using thread Executors...
Add the following code snippets at or after the following line numbers
4 import java.util.concurrent.*;
10 private int pool;
12 this.pool=2;
22 ExecutorService threadExecutor = Executors.newFixedThreadPool(pool);
27 threadExecutor.execute(new Handler(s));
And comment the following 2 lines
27 new Handler(s);
42 new Thread(this).start();
Not lets run the program again on port 5000 and try connecting to it. We have created a pool of 2 threads here. Observer that:
- pool-1-thread-1 & pool-1-thread-2 are the two threads created.
- After two connections to the server, the connections are not rejected, but are queued.
- If a thread becomes available, a connection from the queue is assigned to the thread for processing
- The same two threads are used again and again. No new threads are created.
- Connections which are in queue are not able to process any requests unless they are assigned to a worker thread
Process for testing
console 1: telnet localhost 5000
console 2: telnet localhost 5000
console 3: telnet localhost 5000
console 1: from 1
console 2: from 2
console 3: from 3
console 1: INFO
console 1: 1 quitting
console 1: QUIT
Output for the testing process
jjayant@jayantbox:~/myprogs/java$ java -cp . testPool 5000
Starting Daemon
Thread pool constructor - creating thread
Thread pool - new thread created
Handler constructor - creating thread
pool-1-thread-1 - Handler - new thread created
Handler constructor - creating thread
pool-1-thread-2 - Handler - new thread created
Handler constructor - creating thread
Command = from 1
Command = from 2
Active Connections : 2
Total Connections : 3
Command = 1 quitting
pool-1-thread-1 - Handler - new thread created
Command = from 3
As seen, processing of connection 3 starts only after connection 1 is closed by the client.
The benefits of using a thread executor instead of a normal thread are
a] Overhead of creating and destroying threads is avoided.
b] A queue of threads is created automatically when the no of tasks exceeds the no of threads in the pool. The tasks in queue are automatically executed when the threads become free.
c] A limit (maximum no of threads) could be imposed thus controlling the available resources for delivering better performance.
Tuesday, August 05, 2008
Berkeley DB
I had a chance to look in BDB-XML some time back. But i did not find it that good in performance. BDB on the other hand promises fast storage and retrieval of key-value pairs. For the new-comers, BDB is a embedded database API which can be used to create your own database and fire a defined set of queries on it. BDB does not provide you with an SQL interface or even a database server. All you have is a set of API using which you can write programs to create your own database, populate data into the database, define your own cache and indexes and then use the database to retrieve values for a particular key using your own programs.
With BDB, there is no sql/query optimizer/query parser layer between you and the database.
There are certain types of BDB engines available:
1.] BDB -> the original BDB with c/c++ api. You write programs in c/c++ to create and access your database.
2.] BDB-JAVA -> The java api to BDB. The java api uses JNI in the backend to communicate with the BDB library (written in c/c++).
3.] BDB-JE -> The java edition of the BDB engine. It is a pure java implementation of the BDB engine. It does not use JNI for communication with the system.
4.] BDB-XML -> This is a very sophesticated version of BDB - where you can store XML documents and retrieve documents using any of the keys in the XML Document. You have an XQuery interface where you can fire XML based queries and retrieve results.
The original BDB is ofcourse the fastest.
For a startup, we will take a look at the DPL API of BDB-JAVA. DPL stands for Direct Persistence Layer and is used generally for storing and managing java objects in the database. DPL works best with a static database schema and requires java 1.5.
To create a database using DPL, you generally require an entity class and then create/open a database environment and insert the entity object into the entity store in the database environment. Sounds greek right ?? Lets see an example
Entity Class
import com.sleepycat.persist.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.model.*;
import static com.sleepycat.persist.model.Relationship.*;
@Entity
public class SimpleEntityClass {
// Primary key is pKey
@PrimaryKey
private String pKey;
// Secondary key is the sKey
@SecondaryKey(relate=MANY_TO_ONE)
private String sKey;
public SimpleEntityClass(String pk, String sk)
{
this.pKey = pk;
this.sKey = sk;
}
public void setpKey(String data)
{
pKey = data;
}
public void setsKey(String data)
{
sKey = data;
}
public String getpKey()
{
return pKey;
}
public String getsKey()
{
return sKey;
}
}
Then create a Database Access class which can insert and retrieve Entity objects from the database.
import java.io.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.*;
public class SimpleDA
{
PrimaryIndex pIdx;
SecondaryIndex sIdx;
EntityCursor pcursor;
EntityCursor scursor;
public SimpleDA(EntityStore store) throws Exception
{
pIdx = store.getPrimaryIndex(String.class, SimpleEntityClass.class);
sIdx = store.getSecondaryIndex(pIdx, String.class, "sKey");
}
public void addEntry(String pk, String sk) throws DatabaseException
{
pIdx.put(new SimpleEntityClass(pk, sk));
}
public SimpleEntityClass findByPk(String pk) throws DatabaseException
{
SimpleEntityClass found = pIdx.get(pk);
return found;
}
public ArrayList findBySk(String sk) throws DatabaseException
{
ArrayList ret = new ArrayList();
scursor = sIdx.subIndex(sk).entities();
for(SimpleEntityClass sec1 = scursor.first(); sec1!=null; sec1 = scursor.next())
{
ret.add(sec1);
}
scursor.close();
return ret;
}
}
Create/open the environment to put and retrieve records from the database
import java.io.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.*;
public class SimpleStore
{
private static File envHome = new File("./bdbjava");
private Environment env;
private EntityStore store;
private SimpleDA sda;
public void setup() throws DatabaseException
{
// put all config options here.
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
envConfig.setCacheSize(536870912); //512 MB
envConfig.setCacheCount(2); // 2 caches of 256 MB each
envConfig.setTxnWriteNoSync(true);
envConfig.setInitializeCache(true);
envConfig.setThreaded(true);
envConfig.setInitializeLogging(true);
envConfig.setTransactional(true);
StoreConfig sConfig = new StoreConfig();
sConfig.setAllowCreate(true);
try
{
env = new Environment(envHome, envConfig);
store = new EntityStore(env, "MyDatabaseName", sConfig);
}catch(Exception ex)
{
ex.printStackTrace();
System.exit(-1);
}
}
public SimpleStore()
{
setup();
}
public void putData(String pk, String sk) throws Exception
{
sda = new SimpleDA(store);
sda.addEntry(pk, sk);
}
public void getDataPk(String pk) throws Exception
{
sda = new SimpleDA(store);
SimpleEntityClass sec = sda.findByPk(pk);
System.out.println("pk = "+sec.getpKey()+", sk = "+sec.getsKey());
}
public void getDataSk(String sk) throws Exception
{
sda = new SimpleDA(store);
ArrayList data = sda.findBySk(sk);
for(int x=0; x<data.size(); x++)
{
SimpleEntityClass sec = data.get(x);
System.out.println("pk = "+sec.getpKey()+", sk = "+sec.getsKey());
}
}
public void closeAll() throws Exception
{
store.close();
env.close();
}
public static void main(String[] args)
{
SimpleStore ss = new SimpleStore();
ss.putData("pk1","sk1");
ss.putData("pk2","sk2");
ss.putData("pk3","sk1");
ss.putData("pk4","sk3");
ss.getDataPk("pk1");
ss.getDataSk("sk1");
ss.closeAll();
}
}
So, now you have your own program for creating entities of the type SimpleEntityClass and store them in the database in serialized form. These objects cab be retrieved using primary or secondary keys.
For the relationship between primary and secondary keys please refer to http://www.oracle.com/technology/documentation/berkeley-db/db/java/com/sleepycat/persist/model/Relationship.html
Since you would be storing complete objects instead of just the required data sets, the database size would be relatively high and that would slow down things a bit.
With BDB, there is no sql/query optimizer/query parser layer between you and the database.
There are certain types of BDB engines available:
1.] BDB -> the original BDB with c/c++ api. You write programs in c/c++ to create and access your database.
2.] BDB-JAVA -> The java api to BDB. The java api uses JNI in the backend to communicate with the BDB library (written in c/c++).
3.] BDB-JE -> The java edition of the BDB engine. It is a pure java implementation of the BDB engine. It does not use JNI for communication with the system.
4.] BDB-XML -> This is a very sophesticated version of BDB - where you can store XML documents and retrieve documents using any of the keys in the XML Document. You have an XQuery interface where you can fire XML based queries and retrieve results.
The original BDB is ofcourse the fastest.
For a startup, we will take a look at the DPL API of BDB-JAVA. DPL stands for Direct Persistence Layer and is used generally for storing and managing java objects in the database. DPL works best with a static database schema and requires java 1.5.
To create a database using DPL, you generally require an entity class and then create/open a database environment and insert the entity object into the entity store in the database environment. Sounds greek right ?? Lets see an example
Entity Class
import com.sleepycat.persist.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.model.*;
import static com.sleepycat.persist.model.Relationship.*;
@Entity
public class SimpleEntityClass {
// Primary key is pKey
@PrimaryKey
private String pKey;
// Secondary key is the sKey
@SecondaryKey(relate=MANY_TO_ONE)
private String sKey;
public SimpleEntityClass(String pk, String sk)
{
this.pKey = pk;
this.sKey = sk;
}
public void setpKey(String data)
{
pKey = data;
}
public void setsKey(String data)
{
sKey = data;
}
public String getpKey()
{
return pKey;
}
public String getsKey()
{
return sKey;
}
}
Then create a Database Access class which can insert and retrieve Entity objects from the database.
import java.io.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.*;
public class SimpleDA
{
PrimaryIndex
SecondaryIndex
EntityCursor
EntityCursor
public SimpleDA(EntityStore store) throws Exception
{
pIdx = store.getPrimaryIndex(String.class, SimpleEntityClass.class);
sIdx = store.getSecondaryIndex(pIdx, String.class, "sKey");
}
public void addEntry(String pk, String sk) throws DatabaseException
{
pIdx.put(new SimpleEntityClass(pk, sk));
}
public SimpleEntityClass findByPk(String pk) throws DatabaseException
{
SimpleEntityClass found = pIdx.get(pk);
return found;
}
public ArrayList findBySk(String sk) throws DatabaseException
{
ArrayList ret = new ArrayList();
scursor = sIdx.subIndex(sk).entities();
for(SimpleEntityClass sec1 = scursor.first(); sec1!=null; sec1 = scursor.next())
{
ret.add(sec1);
}
scursor.close();
return ret;
}
}
Create/open the environment to put and retrieve records from the database
import java.io.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.*;
public class SimpleStore
{
private static File envHome = new File("./bdbjava");
private Environment env;
private EntityStore store;
private SimpleDA sda;
public void setup() throws DatabaseException
{
// put all config options here.
EnvironmentConfig envConfig = new EnvironmentConfig();
envConfig.setAllowCreate(true);
envConfig.setCacheSize(536870912); //512 MB
envConfig.setCacheCount(2); // 2 caches of 256 MB each
envConfig.setTxnWriteNoSync(true);
envConfig.setInitializeCache(true);
envConfig.setThreaded(true);
envConfig.setInitializeLogging(true);
envConfig.setTransactional(true);
StoreConfig sConfig = new StoreConfig();
sConfig.setAllowCreate(true);
try
{
env = new Environment(envHome, envConfig);
store = new EntityStore(env, "MyDatabaseName", sConfig);
}catch(Exception ex)
{
ex.printStackTrace();
System.exit(-1);
}
}
public SimpleStore()
{
setup();
}
public void putData(String pk, String sk) throws Exception
{
sda = new SimpleDA(store);
sda.addEntry(pk, sk);
}
public void getDataPk(String pk) throws Exception
{
sda = new SimpleDA(store);
SimpleEntityClass sec = sda.findByPk(pk);
System.out.println("pk = "+sec.getpKey()+", sk = "+sec.getsKey());
}
public void getDataSk(String sk) throws Exception
{
sda = new SimpleDA(store);
ArrayList data = sda.findBySk(sk);
for(int x=0; x<data.size(); x++)
{
SimpleEntityClass sec = data.get(x);
System.out.println("pk = "+sec.getpKey()+", sk = "+sec.getsKey());
}
}
public void closeAll() throws Exception
{
store.close();
env.close();
}
public static void main(String[] args)
{
SimpleStore ss = new SimpleStore();
ss.putData("pk1","sk1");
ss.putData("pk2","sk2");
ss.putData("pk3","sk1");
ss.putData("pk4","sk3");
ss.getDataPk("pk1");
ss.getDataSk("sk1");
ss.closeAll();
}
}
So, now you have your own program for creating entities of the type SimpleEntityClass and store them in the database in serialized form. These objects cab be retrieved using primary or secondary keys.
For the relationship between primary and secondary keys please refer to http://www.oracle.com/technology/documentation/berkeley-db/db/java/com/sleepycat/persist/model/Relationship.html
Since you would be storing complete objects instead of just the required data sets, the database size would be relatively high and that would slow down things a bit.
Friday, August 01, 2008
The Other World...
There is this world where you buy your tickets (be it movie tickets or travel tickets) online. Where you go to the big baazaar an swipe your credit card to make the payment. Where you call up for your car service and the service guys take the car from your place, service it and give it back to you with a bill. You use your credit card for most of the transactions and you rarely deal with cash. You sit in an air-conditioned office and punch a few keys on your computer and you get paid heavily for it. The world of a software related person.
And then there is this other world where everything is done through cash. You rely on the travel agent to get your rail and air tickets. And you get your movie tickets from the movie hall ticket window. You take your car to the service station yourself and stand there and get the service done in front of your eyes. You pay cash all the time and try to avoid getting bills to avoid paying the 12.XX% tax on the total amount. You are not eligible for a credit card cause you dont have any asset to show to the banks.
I had an encounter with the "other world" when i got wood work and painting done in my new flat. I came to know that the margin of profit could be increased for 10% to 60% by simply twisting the quality of materials. I also came to know the fact that the MRP quoted on most materials includes a huge profit margin and that you can actually bargain on the MRP.
When i went to get the first lot of wood i came to know that wood comes from 30Rs/sq ft to more than 100Rs/sq ft. And a increase of 5 Rs/sq ft can make a huge difference in the final cost. Dealing with the workers is a pain. For them you are a fresh,delicious and juicy piece of chicken almost ready for being chopped. And they would love to chop you irrespective of what you think. If you leave them to their will, they would chop you and fight over your pieces. For them even 5 Rs has a meaning. And then there are these people in the local market who only understand cash. If you look at their shop, you wont believe that the shop can have lakhs of rs in cash within itself. There are no guards to guard the shop. And they still have lakhs of cash with them and crores of raw materials just lying there. They dont have a credit card machine cause the bank wont give them any. They dont have credit cards themselves cause they dont have any proof of their huge income. They make lakhs in a month and dont file any income tax returns. They hardly pay any tax. And the fact may be that though they may be serving you, but they may be having more wealth and property than you do.
When i made my first payment of 46000 (cash) to the wood supplier, i filled up my office bag with rupees. I was scared of driving on the road and my hands were literally trembling when i was counting the amount i had to pay. But the shop owner was calm and composed as if it was his daily job. These guys dont know what the internet is or how to use it. All they know is how to make money. It is the other world - the actual world. For us this world is hidden behind a layer of service guys and internet.
And then there is this other world where everything is done through cash. You rely on the travel agent to get your rail and air tickets. And you get your movie tickets from the movie hall ticket window. You take your car to the service station yourself and stand there and get the service done in front of your eyes. You pay cash all the time and try to avoid getting bills to avoid paying the 12.XX% tax on the total amount. You are not eligible for a credit card cause you dont have any asset to show to the banks.
I had an encounter with the "other world" when i got wood work and painting done in my new flat. I came to know that the margin of profit could be increased for 10% to 60% by simply twisting the quality of materials. I also came to know the fact that the MRP quoted on most materials includes a huge profit margin and that you can actually bargain on the MRP.
When i went to get the first lot of wood i came to know that wood comes from 30Rs/sq ft to more than 100Rs/sq ft. And a increase of 5 Rs/sq ft can make a huge difference in the final cost. Dealing with the workers is a pain. For them you are a fresh,delicious and juicy piece of chicken almost ready for being chopped. And they would love to chop you irrespective of what you think. If you leave them to their will, they would chop you and fight over your pieces. For them even 5 Rs has a meaning. And then there are these people in the local market who only understand cash. If you look at their shop, you wont believe that the shop can have lakhs of rs in cash within itself. There are no guards to guard the shop. And they still have lakhs of cash with them and crores of raw materials just lying there. They dont have a credit card machine cause the bank wont give them any. They dont have credit cards themselves cause they dont have any proof of their huge income. They make lakhs in a month and dont file any income tax returns. They hardly pay any tax. And the fact may be that though they may be serving you, but they may be having more wealth and property than you do.
When i made my first payment of 46000 (cash) to the wood supplier, i filled up my office bag with rupees. I was scared of driving on the road and my hands were literally trembling when i was counting the amount i had to pay. But the shop owner was calm and composed as if it was his daily job. These guys dont know what the internet is or how to use it. All they know is how to make money. It is the other world - the actual world. For us this world is hidden behind a layer of service guys and internet.
Subscribe to:
Posts (Atom)