A RAM disk is a portion of RAM which is being used as if it were a disk drive. RAM disks have fixed sizes, and act like regular disk partitions. Their access time is much faster for a RAM disk than for a real, physical disk. However, any data stored on a RAM disk is lost when the system is shut down or powered off. RAM disks can be a great place to store temporary data which needs to be accessed frequently.
For RAM disk to be accessed, the support for the same should be compiled with the linux kernel. As far as i remember Kernel 2.4.* and 2.6.* have RAM disk support compiled in it.
RedHat/Fedora creates 16 ram disks by default. Although they are not usable by default. The size of these ram disks is 4096K (4 MB).
To list available RAM disks use the command - ls -lh /dev/ram*.
To know default size of RAM disk use - dmesg | grep RAMDISK
As it can be seen, using the default configuration only 4 MB RAMDISK can be created. To create larger RAMDISKS you need to configure the kernel parameters so that larger RAMDISKS can be created. How ?? By simply passing the size of the RAMDISK as a parameter during booting of the kernel.
The kernel option for RAMDISK size is: ramdisk_size=<size in kb>.
All you have to do is type this at the boot prompt during booting. Or if you are using some boot loader like lilo or grub, you need to change the boot command in either lilo.conf or grub.conf. For example in grub.conf you would be having something like
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,4)
# kernel /boot/vmlinuz-version ro root=/dev/hda5
# initrd /boot/initrd-version.img
#boot=/dev/hda
default=0
timeout=5
splashimage=(hd0,4)/boot/grub/splash.xpm.gz
hiddenmenu
title Fedora Core (2.6.11-1.1369_FC4)
root (hd0,4)
kernel /boot/vmlinuz-2.6.11-1.1369_FC4 ro root=LABEL=/1 ramdisk_size=512000 quiet
initrd /boot/initrd-2.6.11-1.1369_FC4.img
title Windoze XP
rootnoverify (hd0,1)
chainloader +1
The above configuration file would create ramdisks of size 512MB. There is an upper limit to the max size of RAMDISK that can be created and it is also controlled by the amount of available RAM. So if you have 512MB of RAM, you would not be able to create RAMDISKS of more than 512MB. But depending on the kernel configuration, if you have a 4GB RAM system, you may not be able to create RAMDISK of more than 1GB maybe. You will have to check this limit either with the kernel documents or by trying out different sizes at boot prompt.
Once the RAMDISK size has been configured, all that needs to be done is to create a file system on the RAMDISK and then mount the RAMDISK as a file system directory in linux.
To create an ext2fs file system, the following command is used :
mke2fs -m 0 /dev/ram0
The -m 0 option keeps mke2fs from reserving any space on the file system for the root user. This makes the complete ramdisk space available to any regular user.
To mount the ramdisk :
mkdir /mnt/RAMDISK
mount /dev/ram0 /mnt/RAMDISK
Run mount to check whether the RAMDISK has been mounted.
Now the RAMDISK can be used as a regular directory. You can read, write, delete and modify files on the RAMDISK as if you are working on a regular hard disk. Linux would handle it as if it were handling a regular directory on the disk. The difference between RAMDISK and normal DISK would be invisible to a regular user.
The only problem is that if the system is either rebooted or crashes. In that case, all data in the RAMDISK is lost. You will need to recreate the RAMDISK and redo all that has been done.
The benefit of using RAMDISk is that reads and writes ot the RAMDISK would be extremely fast, since everything is in RAM and not on the DISK. To prevent loss of data, it is advisable to keep a backup of data in RAMDISK on the local HDD.
The commands used to create the RAMDISK and loading of data in RAMDISK can be automated by issueing the command in the rc.local or some other startup script.
More details of effectively creating and using RAMDISKS can be obtained from the kernel documentation.
http://www.vanemery.com/Linux/Ramdisk/ramdisk-kerneldoc.txt
Wednesday, August 23, 2006
Monday, August 21, 2006
trip to kasauli + chail
I have been a bit busy and the blogger was really missing me... anyways, had gone for a trip to kasauli & chail (both are near to Shimla).
Well, initially we were planning to go to jaipur, but we found that due to rains the santuaries would be closed. Moreover, there might be lots of heat in jaipur. And we were planning to go on our bikes. Me on my bullet. So, this got dropped. Next we decided to go to some nearby hillstation. After a lot of research, we decided to visit kasauli and chail. Both are near to each other.
Again the plan of going there on bikes was unsuccessful because one of our group members had a leg injury and it was difficult for him to drive for that long on his bike. It was supposed to be a 6-7 hours/300 kms drive.
Kasauli is around 300 kms from delhi and all we have to do is catch the NH1 to chandigarh and from there to panchkula and then to kasauli/shimla/chail. There is a single road to shimla, on taking a left you reach kasauli and on taking a right you reach chail.
Chail is around 60 kms from kasauli. You have to go down to a small river and then there is a very steep climb up a hill via a narrow road surrounded by greenery and trees. Lovely place to drive through.
We left at around 3 in the morning, gathered at a friends house and then moved on from there in his car. We were able to get on the highway by around 5:30 after filling air and petrol in the car. The highway was good and the drive was great. After a very long time i was able to see the sun rise and we also saw 2 rainbows one above the another on the way.
We reached kasauli around 12 in the afternoon. And we were shocked to see the rates of hotels. We were unable to find anything below 700 for a two bed and nothing below 800 for a 4 bed set. Anyways after lots of research we finalized to stay at R Maidens' some nice hotel with a 4 bed big and decent room. Then had something to eat and went to sleep. Roamed about a bit in the evening and relaxed at night.
Well, the next day we left at around 11 am for chail. Had parathas on the way and enjoyed the hills and the cool and clean air. I was able to see lots of people on bikes and really missed my bullet. At chail, the two things that we visited were - worlds highest cricket pitch and a temple. All the time we were in the clouds. The temple was very peaceful. And the drive wonderful. We came back in the evening and then relaxed.
Next day we went to manki point at kasauli. It was the best place we visited. It is situated inside a military layout, so you could not take any electronic items to that place, neither click photos of the place. We had to leave everything including the our cells, cameras and locking keypad of car inside the car. Had to climb for about 500 meters on a 75 degree slope. On top of th e place, there is a helipad and near that there is a temple. It is a very small place. After reaching the top we were inside the clouds. All we could see was clouds above and below us.
The monkeys over there are very aggressive. During our climb downhill, one of the monkeys took all prasad that we had. Anyways, the visit was good. And the place was worth all the effort.
Later that day, we did some shopping and then relaxed. Next day we left kasauli around 12 pm and were in delhi at around 6:30 pm. Back to the city, heat, pollution and our everyday busy lives.
Well, initially we were planning to go to jaipur, but we found that due to rains the santuaries would be closed. Moreover, there might be lots of heat in jaipur. And we were planning to go on our bikes. Me on my bullet. So, this got dropped. Next we decided to go to some nearby hillstation. After a lot of research, we decided to visit kasauli and chail. Both are near to each other.
Again the plan of going there on bikes was unsuccessful because one of our group members had a leg injury and it was difficult for him to drive for that long on his bike. It was supposed to be a 6-7 hours/300 kms drive.
Kasauli is around 300 kms from delhi and all we have to do is catch the NH1 to chandigarh and from there to panchkula and then to kasauli/shimla/chail. There is a single road to shimla, on taking a left you reach kasauli and on taking a right you reach chail.
Chail is around 60 kms from kasauli. You have to go down to a small river and then there is a very steep climb up a hill via a narrow road surrounded by greenery and trees. Lovely place to drive through.
We left at around 3 in the morning, gathered at a friends house and then moved on from there in his car. We were able to get on the highway by around 5:30 after filling air and petrol in the car. The highway was good and the drive was great. After a very long time i was able to see the sun rise and we also saw 2 rainbows one above the another on the way.
We reached kasauli around 12 in the afternoon. And we were shocked to see the rates of hotels. We were unable to find anything below 700 for a two bed and nothing below 800 for a 4 bed set. Anyways after lots of research we finalized to stay at R Maidens' some nice hotel with a 4 bed big and decent room. Then had something to eat and went to sleep. Roamed about a bit in the evening and relaxed at night.
Well, the next day we left at around 11 am for chail. Had parathas on the way and enjoyed the hills and the cool and clean air. I was able to see lots of people on bikes and really missed my bullet. At chail, the two things that we visited were - worlds highest cricket pitch and a temple. All the time we were in the clouds. The temple was very peaceful. And the drive wonderful. We came back in the evening and then relaxed.
Next day we went to manki point at kasauli. It was the best place we visited. It is situated inside a military layout, so you could not take any electronic items to that place, neither click photos of the place. We had to leave everything including the our cells, cameras and locking keypad of car inside the car. Had to climb for about 500 meters on a 75 degree slope. On top of th e place, there is a helipad and near that there is a temple. It is a very small place. After reaching the top we were inside the clouds. All we could see was clouds above and below us.
The monkeys over there are very aggressive. During our climb downhill, one of the monkeys took all prasad that we had. Anyways, the visit was good. And the place was worth all the effort.
Later that day, we did some shopping and then relaxed. Next day we left kasauli around 12 pm and were in delhi at around 6:30 pm. Back to the city, heat, pollution and our everyday busy lives.
Saturday, August 05, 2006
mysql partitioning
Table partitioning is a concept which allows data to be stored in multiple locations on one or more disks and access to the data is through SQL queries. The way data is stored and retrieved is invisible to the end user.
Mysql allows the user to select the rules based on which data would be spread over multiple partitions in a table. Mysql supports horizontab partitioning whereby the rows of a table are spread across multiple locations. Support for vertical partitioning is not there. So using mysql partitioning, you cannot assign different columns to different physical partitions of a table.
Benefits of partitioning a table :
1. Being able to store more data in one table than can be held on a single disk or filesystem partition. Though current operating systems allow extremely huge files on the disk.
2. Data that loses its usefulness can often be easily be removed from the table by dropping the partition containing only that data. And adding of new data can be facilitated by adding a new partition specially for that data.
3. Some queries can be greatly optimized in virtue of the fact that data satisfying a given WHERE clause can be stored only on one or more partitions, thereby excluding any remaining partitions from the search. Data can also be re-organized to move more frequently accessed data to one partition.
4. Queries involving aggregate functions such as SUM() and COUNT() can easily be parallelized.
5. Greater query throughput can be achieved by spreading data seeks over multiple disks.
I had used partitioning with MYIASM tables and found out that they were very useful. The complete logic of how rows were divided and stored and how they need to be accessed was invisible to me. What happened when i created some partitions for the table is that the same number of .MYI and .MYD files were created. Though the table definition remained in the single .frm file. A separate .PAR file was created for the table which i think was being used to define the logic of how database was being partitioned.
Moreover, an insert on the table locks all the partitions for a small time that is, the time needed by mysql to decide in which partition the record should be put.
The logic using which a table can be partitioned are listed as below:
RANGE partitioning -> Here table partitions are defined based on a range of data to be stored in each partition. For example -
CREATE TABLE abc (
name VARCHAR(100) NOT NULL,
email VARCHAR(40),
dob DATE NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p0 VALUES LESS THAN (1960),
PARTITION p1 VALUES LESS THAN (1970),
PARTITION p2 VALUES LESS THAN (1980),
PARTITION p3 VALUES LESS THAN (1990),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
This would create 5 partitions in the table abc. Partition p0 will contain records whose dob is from 0 to 1959, partition p1 will contain records whose dob is from 1960 to 1969 and so on. In case a row is entered whose dob cannot be accomodated in any of the partitions, an error is thrown and the data is not inserted in the table.
LIST partitioning -> Here table partitions are defined based on a list of data for each partition. Rows which satisfy a criteria list is inserted in that partition.
CREATE TABLE abcs (
name VARCHAR(100),
dob DATE NOT NULL DEFAULT '1979-01-01'
age INT
)
PARTITION BY LIST(age) (
PARTITION p0 VALUES IN (3,5,6,9,17),
PARTITION p1 VALUES IN (1,2,10,11,19,20),
PARTITION p2 VALUES IN (4,12,13,14,18),
PARTITION p3 VALUES IN (7,8,15,16)
);
This would create 4 partitions in the table abcs. Partition p0 would contain records whose age would be 3,5,6,9 or 17. Again if you try to insert a record whose dob is not in any of the partition list created, an error is thrown and the record is not inserted.
LINEAR HASH partitioning -> Here table data is evenly divided between all partitions using some simple functions. for example
CREATE TABLE abc (
name VARCHAR(100),
dob DATE NOT NULL DEFAULT '1970-01-01',
age INT
)
PARTITION BY HASH( YEAR(dob) )
PARTITIONS 4;
Here 4 partitions are created and data is distributed using the following formula
Partition number = MOD(YEAR(dob),number_of_partitions);
So since my date of birth is 1980, my record will be in partition number 0.
LINEAR HASH partitioning -> This is almost similar to hash partitioning except for the fact that the algorithm used to divide data is different. The syntax is also almost same. We use PARTITION BY LINEAR HASH instead of PARTITION BY HASH.
The algorithm used is :
Given an expression expr, the partition in which the record is stored when linear hashing is used is partition number N from among num partitions, where N is derived according to the following algorithm:
The advantage in partitioning by linear hash is that the adding, dropping, merging, and splitting of partitions is made much faster, which can be beneficial when dealing with tables containing extremely large amounts of data. The disadvantage is that data is less likely to be evenly distributed between partitions as compared with the distribution obtained using regular hash partitioning.
KEY partitioning ->
Partitioning by key is similar to partitioning by hash, except that where hash partitioning employs a user-defined expression, the hashing function for key partitioning is supplied by the MySQL server. The syntax used is PARTITION BY KEY instead of PARTITION BY HASH.
In most of the cases either a primary key or an unique key is used to create partitions.
Mysql also supports sub partitioning whereby a partition can be divided into further sub partitions of similar type.
I think i am creating a very long blog for mysql partitioning. However there are a list of syntaxes available on the mysql site to alter, analyze, optimize, rebuild, checking and repairing partitions. Please check http://dev.mysql.com/doc/refman/5.1/en/partitioning.html. It will give a more detailed idea of whatever ranting i have done over here...
I will be signing off from here now. Next blog may or may not contain more info about partitions...
Anyways will keep blogggging...
Mysql allows the user to select the rules based on which data would be spread over multiple partitions in a table. Mysql supports horizontab partitioning whereby the rows of a table are spread across multiple locations. Support for vertical partitioning is not there. So using mysql partitioning, you cannot assign different columns to different physical partitions of a table.
Benefits of partitioning a table :
1. Being able to store more data in one table than can be held on a single disk or filesystem partition. Though current operating systems allow extremely huge files on the disk.
2. Data that loses its usefulness can often be easily be removed from the table by dropping the partition containing only that data. And adding of new data can be facilitated by adding a new partition specially for that data.
3. Some queries can be greatly optimized in virtue of the fact that data satisfying a given WHERE clause can be stored only on one or more partitions, thereby excluding any remaining partitions from the search. Data can also be re-organized to move more frequently accessed data to one partition.
4. Queries involving aggregate functions such as SUM() and COUNT() can easily be parallelized.
5. Greater query throughput can be achieved by spreading data seeks over multiple disks.
I had used partitioning with MYIASM tables and found out that they were very useful. The complete logic of how rows were divided and stored and how they need to be accessed was invisible to me. What happened when i created some partitions for the table is that the same number of .MYI and .MYD files were created. Though the table definition remained in the single .frm file. A separate .PAR file was created for the table which i think was being used to define the logic of how database was being partitioned.
Moreover, an insert on the table locks all the partitions for a small time that is, the time needed by mysql to decide in which partition the record should be put.
The logic using which a table can be partitioned are listed as below:
RANGE partitioning -> Here table partitions are defined based on a range of data to be stored in each partition. For example -
CREATE TABLE abc (
name VARCHAR(100) NOT NULL,
email VARCHAR(40),
dob DATE NOT NULL
)
PARTITION BY RANGE( YEAR(dob) ) (
PARTITION p0 VALUES LESS THAN (1960),
PARTITION p1 VALUES LESS THAN (1970),
PARTITION p2 VALUES LESS THAN (1980),
PARTITION p3 VALUES LESS THAN (1990),
PARTITION p4 VALUES LESS THAN MAXVALUE
);
This would create 5 partitions in the table abc. Partition p0 will contain records whose dob is from 0 to 1959, partition p1 will contain records whose dob is from 1960 to 1969 and so on. In case a row is entered whose dob cannot be accomodated in any of the partitions, an error is thrown and the data is not inserted in the table.
LIST partitioning -> Here table partitions are defined based on a list of data for each partition. Rows which satisfy a criteria list is inserted in that partition.
CREATE TABLE abcs (
name VARCHAR(100),
dob DATE NOT NULL DEFAULT '1979-01-01'
age INT
)
PARTITION BY LIST(age) (
PARTITION p0 VALUES IN (3,5,6,9,17),
PARTITION p1 VALUES IN (1,2,10,11,19,20),
PARTITION p2 VALUES IN (4,12,13,14,18),
PARTITION p3 VALUES IN (7,8,15,16)
);
This would create 4 partitions in the table abcs. Partition p0 would contain records whose age would be 3,5,6,9 or 17. Again if you try to insert a record whose dob is not in any of the partition list created, an error is thrown and the record is not inserted.
LINEAR HASH partitioning -> Here table data is evenly divided between all partitions using some simple functions. for example
CREATE TABLE abc (
name VARCHAR(100),
dob DATE NOT NULL DEFAULT '1970-01-01',
age INT
)
PARTITION BY HASH( YEAR(dob) )
PARTITIONS 4;
Here 4 partitions are created and data is distributed using the following formula
Partition number = MOD(YEAR(dob),number_of_partitions);
So since my date of birth is 1980, my record will be in partition number 0.
LINEAR HASH partitioning -> This is almost similar to hash partitioning except for the fact that the algorithm used to divide data is different. The syntax is also almost same. We use PARTITION BY LINEAR HASH instead of PARTITION BY HASH.
The algorithm used is :
Given an expression expr, the partition in which the record is stored when linear hashing is used is partition number N from among num partitions, where N is derived according to the following algorithm:
- Find the next power of 2 greater than num. We call this value V; it can be calculated as:
V = POWER(2, CEILING(LOG(2, num))) - Set N = F(column_list) & (V - 1).
- While N >= num
{
Set V = CEIL(V / 2)
Set N = N & (V - 1)
}
The advantage in partitioning by linear hash is that the adding, dropping, merging, and splitting of partitions is made much faster, which can be beneficial when dealing with tables containing extremely large amounts of data. The disadvantage is that data is less likely to be evenly distributed between partitions as compared with the distribution obtained using regular hash partitioning.
KEY partitioning ->
Partitioning by key is similar to partitioning by hash, except that where hash partitioning employs a user-defined expression, the hashing function for key partitioning is supplied by the MySQL server. The syntax used is PARTITION BY KEY instead of PARTITION BY HASH.
In most of the cases either a primary key or an unique key is used to create partitions.
Mysql also supports sub partitioning whereby a partition can be divided into further sub partitions of similar type.
I think i am creating a very long blog for mysql partitioning. However there are a list of syntaxes available on the mysql site to alter, analyze, optimize, rebuild, checking and repairing partitions. Please check http://dev.mysql.com/doc/refman/5.1/en/partitioning.html. It will give a more detailed idea of whatever ranting i have done over here...
I will be signing off from here now. Next blog may or may not contain more info about partitions...
Anyways will keep blogggging...
Subscribe to:
Posts (Atom)