Trying out windows 7

So, you have windows xp and linux (preferably ubuntu) installed on your system. And you did not install windows vista because it has issues with running some games and also because it is a resource hog. It chews on your resources until none is left. Well, you saw windows 7, you read its reviews and you figured out that maybe you should try windows 7. Here is a how-to of how i went about installing windows 7 and the moments of despair and surprise associated with it.

First of all, you have to download windows 7 from the microsoft's site. I was surprised that microsoft - for the first time in the history of mankind is giving out windows 7 OS for free (even if it is for a year only). Go ahead and download Windows 7. Select the version you want to download 32 bit or 64 bit. Preferably you should go with 32 bit if dont want to worry much about compatibility issues. But if you feel very adventurous, you can also try out the 64 bit version. The 32 bit is of around 2.5 GB and the 64 bit is of around 3 GB. I was surprised by the download speed. I was ablt to download it in an hour or so.

Next, get a DVD-R, and burn the image on DVD-R. But before you proceed to installing windows 7, you need to create some space on your already filled up drive. I have a 500 GB drive with 250 GB dedicated to dumps of games and movies and songs and pics. Out of the remaining 250 GB, i had 100 GB dedicated to windows xp and 150 GB dedicated to linux (2 drives - 100 GB /home and 50 GB /). So, to create more space i decided to shrink my linux drives to 70 GB /home and 30 GB / making around 50 GB free space for windows 7.

To go about resizing partitions, i first decided to use partition magic. But then the geek within me protested. Why should i go ahead and find a crack for a paid partition editor when i can use something like gparted (in linux) to do the same work. I popped in a live cd of kubuntu 8.10 (thinking about it now - i should have created a live cd of ubuntu 9.04). After boot up, I found out that kubuntu does not ship with gparted installed. Hell, so i did a "sudo apt-get install gparted" to install gparted on the virtual drive. It took around 10-15 minutes on my 256 KB connection.

After opening up gparted i simply resized the required partitions and applied the changed. Resizing a partition is an easy job. But moving partitions around is a heavy job. So make changes in such a manner that moving parititons around should not be necessary. I made the mistake of moving my bigger partition (70 GB) and i realized it later. It took around 1.5 hours to do this. And it is always advisable to backup your important data on an external drive before doing this. In case you screw up.

By luck every thing went well and later i was able to boot up both in windows xp & ubuntu.

Now was the time to start installation of windows 7. I popped in the DVD and the main screen came up. Remember to write down the product key that you got during downloading the iso. Windows still punishes you severely if you dont write down the product key. It asks the product key in the middle of installation and if you have forgotten to note down the product key, you will have to do all the work again.

The installation of windows 7 took around 45 minutes. The bad thing is that at times there is no activity and I was confused whether anything is happening or not. Remember to wait and watch. Basically there are 3 steps in installation. Remember to choose custom installation and the free space on your drive for installation. I was surprised that installation started in a flash after i selected the drive. I was hoping to get an installation summary and an option of correcting any mistake i had done. So be-ware, a wrong click and kill your system. Remember to remove the DVD during the first boot - it does not prompt you to do so, but if you dont, you would be starting the installation again. After 2-3 reboots your system is ready and you will get a boot screen (black and white) with option for "earlier version of windows" or "windows 7".

Now all you need to do is restore your grub boot menu. Again pop in your ubuntu live cd and boot up ubuntu. Open a terminal and type.

sudo grub
grub> find /boot/grub/stage1
If you get error 15 "file not found" try
grub> find /grub/stage1
You will get something like (hdX,Y). Do
grub> root (hdX,Y)
grub> setup (hd0)
grub> quit

Now simply reboot and you should be getting your grub boot menu back. Select windows and get the boot menu for windows xp / windows 7.

Whew!!!!

creating opportunity

If opportunity does not knock on the door, open the door and drag the opportunity inside...

mount linux drive on windows

Earlier there used to be a software known as explore2fs which used to allow the users to scan all file-systems in read-only mode and copy files from the linux drive to windows. I had used it long time ago...

Now-a-days things have changed quite a lot. When i googled for mount ext3 file system on windows, i got tons of links and tons of tools. For some time i was confused how to go about all of it.

But then after going thru all the tools, i came across something known as ext2fsd. I got the 0.46 version downloaded - only 973KB and installed it.

Once you start the ext2fsd volume manager, you could see all the file systems on your disk. All you have got to do is

1. right click on the drive you want to mount and assign it a drive letter.
2. go to the ext2 management and choose the drive letter as mount point for fixed boot
3. ask the program to automatically mount the drive on boot
4. enable ext2fsd to auto-start during boot.

A simple reboot and i could see my linux partition as a drive on the windows explorer. No need to copy files to read them (like i used to do when i used explore2fs). It runs like a charm...

I still have to look at how to go about mounting ext4 file system on windows. Though ext4 is not that common, but it should become common soon...

The other Stall

This could happen to you tooo.........

I was barely sitting down when I heard a voice from the other stall saying:
'Hi, how are you?'

I'm not the type to start a conversation in the restroom. I don't know what got into me, but I answered, somewhat embarrassed, 'Doin' just fine!'

And the other person says:
'So what are you up to?'

What kind of question is that? At that point, I'm thinking this is too bizarre so I say:
'Uhhh, I'm like you, just sitting here.'

At this point I am just trying to get out as fast as I can when I hear another question:
'Can I come over?'

Ok, this question is just too weird for me. I figured I could politely end the conversation. I say:
'No........I'm a little busy right now!!!'

Then I hear the person say, nervously:
'Listen, I'll have to call you back. There's an idiot in the other stall who keeps answering all my questions!!'

Avoid using the mobile when you are answering a nature's call

How facebook stores billions of photos?

The new photo infrastructure merges the photo serving tier and storage tier into one physical tier. It implements a HTTP based photo server which stores photos in a generic object store called Haystack. The main requirement for the new tier was to eliminate any unnecessary metadata overhead for photo read operations, so that each read I/O operation was only reading actual photo data (instead of filesystem metadata). Haystack can be broken down into these functional layers -

* HTTP server
* Photo Store
* Haystack Object Store
* Filesystem
* Storage

In the following sections we look closely at each of the functional layers from the bottom up.

Storage

Haystack is deployed on top of commodity storage blades. The typical hardware configuration of a 2U storage blade is –

* 2 x quad-core CPUs
* 16GB – 32GB memory
* hardware raid controller with 256MB – 512MB of NVRAM cache
* 12+ 1TB SATA drives

Each storage blade provides around 10TB of usable space, configured as a RAID-6 partition managed by the hardware RAID controller. RAID-6 provides adequate redundancy and excellent read performance while keeping the storage cost down. The poor write performance is partially mitigated by the RAID controller NVRAM write-back cache. Since the reads are mostly random, the NVRAM cache is fully reserved for writes. The disk caches are disabled in order to guarantee data consistency in the event of a crash or a power loss.

Filesystem

Haystack object stores are implemented on top of files stored in a single filesystem created on top of the 10TB volume.

Photo read requests result in read() system calls at known offsets in these files, but in order to execute the reads, the filesystem must first locate the data on the actual physical volume. Each file in the filesystem is represented by a structure called an inode which contains a block map that maps the logical file offset to the physical block offset on the physical volume. For large files, the block map can be quite large depending on the type of the filesystem in use.

Block based filesystems maintain mappings for each logical block, and for large files, this information will not typically fit into the cached inode and is stored in indirect address blocks instead, which must be traversed in order to read the data for a file. There can be several layers of indirection, so a single read could result in several I/Os depending on whether or not the indirect address blocks are cached.

Extent based filesystems maintain mappings only for contiguous ranges of blocks (extents). A block map for a contiguous large file could consist of only one extent which would fit in the inode itself. However, if the file is severely fragmented and its blocks are not contiguous on the underlying volume, its block map can grow large as well. With extent based filesystems, fragmentation can be mitigated by aggressively allocating a large chunk of space whenever growing the physical file.

Currently, the filesystem of choice is XFS, an extent based filesystem providing efficient file preallocation.

Haystack Object Store

Haystack is a simple log structured (append-only) object store containing needles representing the stored objects. A Haystack consists of two files – the actual haystack store file containing the needles, plus an index file. The following figure shows the layout of the haystack store file:


The first 8KB of the haystack store is occupied by the superblock. Immediately following the superblock are needles, with each needle consisting of a header, the data, and a footer:


A needle is uniquely identified by its <Offset, Key, Alternate Key, Cookie> tuple, where the offset is the needle offset in the haystack store. Haystack doesn’t put any restriction on the values of the keys, and there can be needles with duplicate keys. Following figure shows the layout of the index file -





There is a corresponding index record for each needle in the haystack store file, and the order of the needle index records must match the order of the associated needles in the haystack store file. The index file provides the minimal metadata required to locate a particular needle in the haystack store file. Loading and organizing index records into a data structure for efficient lookup is the responsibility of the Haystack application (Photo Store in our case). The index file is not critical, as it can be rebuilt from the haystack store file if required. The main purpose of the index is to allow quick loading of the needle metadata into memory without traversing the larger Haystack store file, since the index is usually less than 1% the size of the store file.

Haystack Write Operation

A Haystack write operation synchronously appends new needles to the haystack store file. After the needles are committed to the larger Haystack store file, the corresponding index records are then written to the index file. Since the index file is not critical, the index records are written asynchronously for faster performance.

The index file is also periodically flushed to the underlying storage to limit the extent of the recovery operations caused by hardware failures. In the case of a crash or a sudden power loss, the haystack recovery process discards any partial needles in the store and truncates the haystack store file to the last valid needle. Next, it writes missing index records for any trailing orphan needles at the end of the haystack store file.

Haystack doesn’t allow overwrite of an existing needle offset, so if a needle’s data needs to be modified, a new version of it must be written using the same tuple. Applications can then assume that among the needles with duplicate keys, the one with the largest offset is the most recent one.

Haystack Read Operation

The parameters passed to the haystack read operation include the needle offset, key, alternate key, cookie and the data size. Haystack then adds the header and footer lengths to the data size and reads the whole needle from the file. The read operation succeeds only if the key, alternate key and cookie match the ones passed as arguments, if the data passes checksum validation, and if the needle has not been previously deleted (see below).

Haystack Delete Operation

The delete operation is simple – it marks the needle in the haystack store as deleted by setting a “deleted” bit in the flags field of the needle. However, the associated index record is not modified in any way so an application could end up referencing a deleted needle. A read operation for such a needle will see the “deleted” flag and fail the operation with an appropriate error. The space of a deleted needle is not reclaimed in any way. The only way to reclaim space from deleted needles is to compact the haystack (see below).

Photo Store Server.

Photo Store Server is responsible for accepting HTTP requests and translating them to the corresponding Haystack store operations. In order to minimize the number of I/Os required to retrieve photos, the server keeps an in-memory index of all photo offsets in the haystack store file. At startup, the server reads the haystack index file and populates the in-memory index. With hundreds of millions of photos per node (and the number will only grow with larger capacity drives), we need to make sure that the index will fit into the available memory. This is achieved by keeping a minimal amount of metadata in memory, just the information required to locate the images.

When a user uploads a photo, it is assigned a unique 64-bit id. The photo is then scaled down to 4 different sizes. Each scaled image has the same random cookie and 64-bit key, and the logical image size (large, medium, small, thumbnail) is stored in the alternate key. The upload server then calls the photo store server to store all four images in the Haystack.

The in-memory index keeps the following information for each photo:

Haystack uses the open source Google sparse hash data structure to keep the in-memory index small, since it only has 2 bits of overhead per entry.

Photo Store Write/Modify Operation

A write operation writes photos to the haystack and updates the in-memory index with the new entries. If the index already contains records with the same keys then this is a modification of existing photos and only the index records offsets are modified to reflect the location of the new images in the haystack store file. Photo store always assumes that if there are duplicate images (images with the same key) it is the one stored at a larger offset which is valid.

Photo Store Read Operation

The parameters passed to a read operation include haystack id and a photo key, size and cookie. The server performs a lookup in the in-memory index based on the photo key and retrieves the offset of the needle containing the requested image. If found it calls the haystack read operation to get the image. As noted above haystack delete operation doesn’t update the haystack index file record. Therefore a freshly populated in-memory index can contain stale entries for the previously deleted photos. Read of a previously deleted photo will fail and the in-memory index is updated to reflect that by setting the offset of the particular image to zero.

Photo Store Delete Operation

After calling the haystack delete operation the in-memory index is updated by setting the image offset to zero signifying that the particular image has been deleted.

Compaction

Compaction is an online operation which reclaims the space used by the deleted and duplicate needles (needles with the same key). It creates a new haystack by copying needles while skipping any duplicate or deleted entries. Once done it swaps the files and in-memory structures.

HTTP Server

The HTTP framework we use is the simple evhttp server provided with the open source libevent library. We use multiple threads, with each thread being able to serve a single HTTP request at a time. Because our workload is mostly I/O bound, the performance of the HTTP server is not critical.

Summary

Haystack presents a generic HTTP-based object store containing needles that map to stored opaque objects. Storing photos as needles in the haystack eliminates the metadata overhead by aggregating hundreds of thousands of images in a single haystack store file. This keeps the metadata overhead very small and allows us to store each needle’s location in the store file in an in-memory index. This allows retrieval of an image’s data in a minimal number of I/O operations, eliminating all unnecessary metadata overhead.

Prince Of Persia

Screen shot from Prince Of Persia 2008

Oblivion

Screenshot from Oblivion - The Elder Scrolls IV

My hunt for air conditioner

It was monday in mid april. We were running our cooler at full speed. But it was still throwing out hot air. I tried sleeping but the heat was too much to bear. Even the bed was hot and the pillows were wet with sweat. That was the time when i decided that i should be going for an ac. Air conditioning is essential for a good night sleep in india.

My understanding was that i would get 2 ACs - one for me and another for my parents and it should cost between 25K to 30K max. I opted out of split ACs because, they require a lot of work for installation and cost a lot as well. Window ACs are relatively cheaper and since I had window AC space in all my rooms, it would be easy to install.

My first problem was to figure out the tonnage of AC that i should go for. Calculators are available - if you search for "AC Calculator" on google. Most AC calculators provide the amount of BTU required for specified room size. BTU stands for British Thermal Unit - something used to calculate the amount of energy. Well most calculators say that for a room of size 14 x 11 with sunlight should require an AC of around 10,000 BTU. And another room of size 10 x 12 should require around 8,000 BTU. A general 1 Ton AC outputs around 12,000 BTU & a 1.5 Ton AC outputs around 18,000 BTU.

So all this technical babble tells me that i should go for a 1 Ton AC in my smaller room and a 1.5 Ton AC in my bigger room. Though the AC tonnage calculator by LG at http://www.in.lge.com/Support/rac_calculator.aspx says that you should always go for a 1.5 Ton AC no matter what the room size is.

Next and the most important thing is to decide the amount of electricity an AC would consume. There are ACs which consume less electricity than the others. This fact has a formula and the formula is known as EER (Energy Efficiency Ratio). EER is simple to calculate. Just divide the BTU with the Watts. So if you take an AC with 12,000 BTU and which consumes 1300 watts - the EER for that AC would be 12000/1300 = 9.23. Good ACs should have an EER of > 10.0. In India the calculation is done differently and a star rating is calculated and put on all the ACs. So a 3* AC is more efficient than a 2* AC, and the cose of a 3* AC is more than a 2* AC.

After a lot of research and armed with all this information i started out on a hot sunny saturday to get the ACs. First stop was at Chroma Electronics store. They dont know anything about EER or Wattage or Tonnage calculation. The range of ACs available were only Voltas and Hitachi. All they knew were about the stars. A voltas 1* 1.5 ton would cost around 15K. Same thing with 2* would cost 18K and 3* would cost 21K. A nearby store was promoting haier window ACs 1.5 Ton with EER of around 10.3 at 17K. I have no idea how good they are.

Our next stop was sector 18, Noida. Just in front of centerstage mall, noida there is a 3 storey shop which has almost all brands of AC. Again they were promoting Voltas, LG, Samsung, Hitachi. They had Onida and OGeneral as well. But the models were very few and they looked like no one was purchasing them. Even godrej sells ACs. I was surprised. The market is full of 1* AC at 15K. No one was willing to compromise on the price. Imagine a world without discounts. Later after roaming around for 3-4 hours i was able to figure out that major players in the market were Hitachi, LG, Voltas and Carrier. You could also get ACs from Samsung, Electrolux, Whirlpool, Onida, Ogeneral, Haier, Godrej, videocon - but they were not as popular. People generally asked for LG and took a 1* AC.

I was very confused. I realized that i had not done enough research. The sales executive were terrible. In the same shop two sales executive were selling different products. One says that whirlpool is the best AC - no problems has been reported. Other salu in the same shop would say that Voltas is the best AC. It is from TATA. Whirlpool and all are just crap. I also re-realized that salus are meant to sell. It is said that a sales person should be able to sell an AC to a person living on the north pole. You can never trust a salu.

Everybody was tired and exhausted due to my research. We - who sit in the AC all day long and type on the key-board to get our monthly paycheques, have all turned soft. Even an hour of walking we run out of breath. And i have been dragging everyone for about 3 hours in the sun. I realized that i should take a decision without proper research - else everyone would be disheartened - 3 hours of walking in the sun and still no result would be frustrating.

Then i called up a friend of mine and he told me that if you are undecided, you can close your eyes and purchase a Hitachi. It is around 4 thousand costlier than other ACs but it is very efficient and in a year's time it should be saving enough to make up for the extra cost. And in the long run it is more cost efficient. I then remembered my debate of whether to purchase a diesel or a petrol car. I had decided to purchase a petrol car which is costlier than a diesel car - but whose running cost is high because petrol is costlier than diesel. And though i did not regret my decision - but i still debate the same fact. I had gone for initial low cost and high cost of running with my car. So, i should go for initial high cost and low running cost. Maybe it will be beneficial in the end. Bingo - i made the decision to take a Hitachi AC. 1 Ton with EER of 10.6 costed 21.5K and 1.5 Ton with EER of 10.6 costed 25K. Whereas i was getting 1.5 Ton Voltas with 3* - EER of around 10 for 21K.

Remember that An AC has an additional cost of installation which is around 500/- and you have to have a voltage stabilizer - which starts from 2.2K and goes up to 4-5K. It is better to go for a voltage stabilizer with copper winding.

Later, the next day i sat down and did my research and found out that AC has 3 brands to look out for - Voltas, Carrier and Hitachi - for budget, mid-range and the best AC respectively. Carrier is a very good option for those who are looking at better efficiency and low power consumption in mid-range. Maybe i should have bought a carrier AC and saved myself some money. Anyways, the AC is installed and running successfully.

Linux to the rescue

memcached replication

Wow... finally a solution that provides replication in memcache - repcached.

You can have a look at it repcached.lab.klab.org.

They provide two types of packages

1. a pached memcache source, which can be directly compiled.
2. a patch which can be applied to the memcache source and then compiled.

So, i downloaded the memcached-(version)-repcached-(version).tar.gz source and simply compiled it.

./configure --enable-replication
make
sudo make install

Note : When you enable replication, you cannot do --enable-threads.

I started two instances of memcached on ports 11211 & 11222

jayant@gamegeek:~/php$ memcached -p 11211 -m 64 -x 127.0.0.1 -v
replication: connect (peer=127.0.0.1:11212)
replication: marugoto copying
replication: close
replication: listen

jayant@gamegeek:~/php$ memcached -p 11222 -m 64 -x 127.0.0.1 -v
replication: connect (peer=127.0.0.1:11212)
replication: marugoto copying
replication: start


Now set and get a value on instance on port 11211

jayant@gamegeek:~$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
set hello 0 0 5
world
STORED
get hello
VALUE hello 0 5
world
END


Connect to port 11222 and try getting this value

jayant@gamegeek:~$ telnet localhost 11222
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get hello
VALUE hello 0 5
world
END


Try the reverse as well

On 11222
<---snip-->
set key 0 0 5
myval
STORED
get key
VALUE key 0 5
myval
END
<---snip-->

On 11212
<---snip-->
get key
VALUE key 0 5
myval
END
<---snip-->

Suppose the master goes down (in this case lets assume that the memcached on port 11211 goes down). So, we redirect all traffic on port 11222. But later when memcached on port 11211 comes up, the data should be automatically replicated on the new instance. Lets kill the memcache on port 11211 and restart it

On port 11211

Killed
jayant@gamegeek:~/php$ memcached -p 11211 -m 64 -x 127.0.0.1 -v
replication: connect (peer=127.0.0.1:11212)
replication: marugoto copying
replication: start


On port 11222

<---snip-->
replication: close
replication: listen
replication: accept
replication: marugoto start
replication: marugoto 2
replication: marugoto owari
<---snip-->

Lets see if the data has been replicated on port 11211

jayant@gamegeek:~$ telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
get hello
VALUE hello 0 5
world
END
get key
VALUE key 0 5
myval
END


Bingo...
Please share your experience if you have tried it on a live scenario with large number of sets and gets.