Tuesday, January 26, 2016

Build your own in house weather station

We all use a smartphone and most of the smart phones generally have a weather app. Which shows the weather for your city. But do you wonder what would be the weather like inside your house or inside your room? The "in house weather station" monitors the weather inside your house. With its help, you can get a report of what the weather is outside and compare it with the weather inside. Here we are monitoring only two parameters - temerature and humidity. After all the hard work, you would get an output similar to this

Tools :
  • Raspberry pi 2 with raspbian installed
  • DHT22 sensor
  • 10k resistor
  • Male to female jumper wires
  • Breadboard
  • Wires for making breadboard connections
I would not go in depth on how to setup the DHT22 sensor. Here are the instructions that I followed.

http://www.rototron.info/dht22-tutorial-for-raspberry-pi/

I was stuck with the resistor piece as I did not have a 10K resistor. So I added two 6k resistors in series, taking the total resistance to 12k (6k+6k).

Here is how my setup looks

W.r.t the software, I tried using the Adafruit code, but it felt more comfortable using the pigpio code after running the pigpiod daemon. I am still getting some errors when i call the s.cancel() and pi.stop() functions. But the rest of the code works.

To get the external temperature and humidity, I signed up with openweather and got their api. There is a python library pyowm which provides an implementation for openweather. So, I got the humidity and temperature of my city from openweather. And i pushed the data - external humidity & temperature and internal humidity & temperature into elasticsearch which I was running on my raspberry pi.

Looking at the limited resources available on the pi, both in terms of disk and memory, I did some tweaking in elasticsearch, so that It did not hog down my raspberry pi.

I restricted the heap size of elasticsearch to 256MB by adding the following options in the ./bin/elasticsearch script.

ES_HEAP_SIZE=256m
ES_JAVA_OPTS=-server

Since, I had to access my elasicsearch via the raspberry pi's ip address, i changed the host to bind to the ip address instead of localhost. Also, looking at the disk constraints, I limited the number of shards & replicas to 1. Here are the configuration lines in ./config/elasticsearch.yml

network.host: 192.168.1.51
index.number_of_shards: 1
index.number_of_replicas: 1

Here is the script which pulls data from openweather and gets data from my DHT22 sensor and pushes it into elasticsearch.



The interesting thing to note in the graph is that though there is change in the external temperature and humidity, the internal temperature and humidity does not change much.

Saturday, November 28, 2015

Use your smartphone as a GUI display for your pi

With a very simple hack, we can use our smartphone as a GUI display for your raspberry pi. This can work over remotely and there is not HDMI or any other connector required to connect your smartphone to your pi.

There are a few apps that you would require on your smartphone. Download and install

Xserver XSDL
Connectbot

Xserver XSDL is a xserver which can be run on your smartphone. Whereas connectbot is a simple ssh client for connecting to your raspberry pi via ssh. The trick here is to run the Xserver on your smartphone. Ssh to your pi and tell your pi to forward the display to the Xserver running on your smartphone.

Let us see the configurations that need to be verified on the pi. Ssh to your raspberry pi over the network and check the following configuration options in these files.

/etc/ssh/sshd_config

AllowTcpForwarding yes
X11Forwarding yes
X11DisplayOffset 10



/etc/ssh/ssh_config

ForwardX11 yes
ForwardX11Trusted yes

If these configuration options are commented out, simply uncomment them. Else add them and restart your ssh server on your pi.

$ sudo service ssh restart

Now start the app Xserver XSDL on your smart phone. It may download a set of fonts and will then ask you to run some commands on your linux PC. The Xserver starts running on port 6000. Now, we need to forward the display of pi to our smartphone.

Go back to the ssh connection on your smartphone and run these commands.

$ export DISPLAY=192.168.1.37:0           
$ export PULSE_SERVER=tcp:192.168.1.37:6000


My smartphone is running on the ip 192.168.1.37. And i am forwarding the display directly to the Xserver running on port 6000.
This is insecure as the port forwarding is not happening over ssh but directly from the pi to the smartphone. But if you are on a local network, it should not matter.

Now simply start your lxde desktop environment on the ssh connection. Execute the following command on the ssh connection.

$ startlxde

And bingo...


It is not very comfortable for working as the mouse pointer is difficult to catch and click. But if you have an old smartphone or tablet lying around, you can use it as a display - running a smart photo frame or a hanging smart calendar.

Monday, November 09, 2015

Script to update godaddy dns A record

Recently, I have setup my raspberry pi as a download machine. The aim was to be able to access the machine from anywhere and be able to add files to download. Now, since the machine is connected via airtel broadband, it is bound to go down and come up as and when the electricity in my location does.

Everytime, when the machine came up, i would be having a different public ip address to access the machine as my ADSL modem+router would also go down.

At first, i setup a cron to be run at boot time which would give me the public ip address.

The corresponding cron entry is
@reboot /home/pi/progs/godaddy.py

I also created a dns entry to access the machine. And then wrote a script to update the dns record in godaddy whenever the machine reboots. Here is the script.

Now whenever my pi reboots, it checks and updates the corresponding DNS record at godaddy. And i can ssh my machine from anywhere.

Tuesday, January 20, 2015

Ecommerce review : Amazon.in shopping experience

I use ecommerce extensively. All my stuff - from clothes to shoes to electronics to groceries are delivered to my doorstep. The good point about ecommerce is that it saves you from the headache of driving down to the store, parking, standing in a queue to pay for your purchase, and hauling the stuff back to your place. And I use multiple shopping sites for getting my stuff:
Flipkart for books & electronics
Firstcry for diapers & other stuff for kids
Zovi, jabong, myntra in order of preference for clothes and shoes
Peopleeasy for groceries.

I try to avoid shopping on amazon. Reason being that the delivery times are too long. Also there is no quality control from amazon. My last order from amazon was a pair of school shoes for my kid which were defective. Since it was not "fulfilled by amazon" the only option I had was to send it back by a courier and wait for 15-30 days for a refund or go to a nearby cobbler and get the defect fixed. I chose the later. It costed me an additional amount, but saved me from the hassles of refund and re-order.

After that experience, I decided that the only option why I would order from amazon was when the product will be "amazon fulfilled".

I was looking for a coffee maker and found one on flipkart. But flipkart will not deliver it to my pin code. I have never tried snapdeal or paytm but have heard first hand horror stories from people who have used it. The only option was amazon which had the exact model that I was looking for and that too with the "amazon fulfilled" label. The delivery time was 10 days, but still I went ahead and ordered. I still cannot figure out why it takes 10 days for amazon to deliver while flipkart delivers in 3-4 days. And zovi & jabong deliver in 1-2 days. Zovi delivers from bangalore and that too in 2 days max.

When I got a mail that my package has been dispatched by india post, I was a bit astonished. What role does india post play in amazon fulfilled orders. I believe it is the cheapest option. But has service and reliability issues. I kept a track of the package on india post web site as amazon did not update the tracking details on its website frequently.

Finally after 7 days the package arrived in the post office nearest to me. I was hoping that I will get the order the next day. But what happened the next day shook my perception of e-commerce.

I got a call from the post office asking me to pick up the package from there. I was like - what??? And the person on the other end said that it is too big for him to deliver and cut off the phone. I connected up with amazon customer support and was given the assurance that I will get the package on my doorstep next day. Here is the actual conversation:

10:00 AM IST Amazon(Amazon): Thanks for contacting Amazon.in customer service. 
My name is Vijay. 
10:00 AM IST Jayant kumar : hi Vijay, 
10:00 AM IST Amazon(Amazon): Hi! Jayant 
10:01 AM IST Jayant kumar : i got a call from india post telling me that my package is at their facility and i have to pick it up ...
If i have to go out pick up a package, then why order online, wait for 15 days and then pick it up
i can simply go out and buy it from a store
10:02 AM IST Jayant kumar : is this what you call e-shopping ?
10:02 AM IST Amazon(Amazon): I really do apologize for the inconvenience this may have caused you.
10:02 AM IST Jayant kumar : Also, please let me know why this order is being delivered by india post, instead of amazon ?
10:02 AM IST Amazon(Amazon): Let me check what best I can do for you.
10:03 AM IST Amazon(Amazon): I will surely let you know about the same.
10:03 AM IST Jayant kumar : i ordered it because it said that this will be fulfilled by amazon
10:04 AM IST Amazon(Amazon): Jayant, I really appreciate for doing business with us.
10:04 AM IST Jayant kumar : and not india post - the worst courier service in the country
10:05 AM IST Amazon(Amazon): I have contacted the courier partner and have instructed them to deliver to your door step.
10:05 AM IST Jayant kumar : I was told that since the package is "big" it may be broken if delivered to my door step
in case the package is broken, i hope amazon will take it back...
10:06 AM IST Amazon(Amazon): And regarding the courier partner, it is automatically selected by the system and we do not have the option to select the same. 
But we try to dispatch through Amazon service as much as we can.
10:07 AM IST Jayant kumar : Since this package is fulfilled by amazon, please explain how?
10:07 AM IST Amazon(Amazon): Related to broken item, we assure you it will be taken back in such scenarios and will issue a full refund.
10:08 AM IST Jayant kumar : so, fulfilled by amazon does not mean amazon delivery service ?
10:09 AM IST Amazon(Amazon): Amazon Fulfilled guarantees the product to be in a perfect condition and also assures if any defects found customer can return it back within the return window and we will issue a full refund.
10:10 AM IST Amazon(Amazon): But, as I have stated earlier the courier does get selected automatically by the system.
10:10 AM IST Jayant kumar : ok
so, will this be delivered today ?
10:11 AM IST Jayant kumar : since it is at the hub nearest to my place?
10:11 AM IST Amazon(Amazon): I have contacted the courier for the delivery as soon as possible.
10:12 AM IST Jayant kumar : thanks
10:12 AM IST Amazon(Amazon): You are most welcome Jayant.
Is there anything else I may assist you with?
10:13 AM IST Jayant kumar : no
10:13 AM IST Amazon(Amazon): It was a pleasure assisting a valued customer as you.
Hope to see you soon.
10:13 AM IST Jayant kumar : looking forward to ordering more with confidence from amazon
10:13 AM IST Amazon(Amazon): Thank you for contacting Amazon Customer Service.
Have a Wonderful time and day ahead.
10:14 AM IST Amazon(Amazon): I assure, you will not face any such issue in the future. 

Note the final line - I was hoping that this problem is now resolved and was looking forward to the package.

What happened the next day is that in the morning, I got a call from the postman. Again asking me to come and collect the package. It seems that amazon customer service did not take any action on the issue raised by me and has given me false assurance. The two options available to me were to either cancel the order and wait for refund and go out and purchase from the market or go ahead and pick the package from india post. I chose the later - being the easier option of the two. The reason given by the postman was genuine that he cannot deliver a big package on his cycle to which I could not come up with a counter argument.

I was told that the post office opens from 10 am to 5 pm and knowing how government officials work - allowing them their tea, lunch and other breaks, I planned to visit the post office around 3 pm. When they would be done with their lunch and after lunch chit chat. When I reached there, I had to enquire around 4-5 people but was not able to trace where my package was. Everyone was busy doing their own work. So, I called up the postman and told him to connect me to the person who can locate and hand over the package to me. Finally after asking everyone's name and making the person talk with the postman, I was able to take delivery of the package.

So, there is me now carrying 2 big boxes in front of me from the post office to the car and from the car to my flat. All the while people staring at the amazon.in label on the boxes I am carrying and trying to figure if I am the guy who delivers for amazon. They must be figuring the ad of amazon where the delivery guy delivers the package with a smile. I, on the other hand was only concerned if I would encounter anyone who has faced similar fate by amazon and try to engage my as an amazon employee.

By luck the products were ok. I have no idea what will happen if I had to opt for an exchange or refund. If delivery is suoh a hassle, refund / exchange can be non existent. Also I cannot figure out how amazon plans to compete with other sites. When even small sites like zovi outsmart amazon on delivery. I would not be ordering from amazon till I hear some positive reviews.

Tuesday, September 09, 2014

gitlab hooks and disabling forced push

For people who do not know much about gitlab, it is a "github" equivalent with the possibility of deployment in a local network. Which means, that if your organization is not comfortable of pushing code to an external website like github, you can get all the features that github provides in your local environment.

The installation of gitlab is not an easy process. If you have an ubuntu-type system, you can get a .deb file and simply install it. Else installation is a very tedius task.

After installation and deployment, the first task that you would think about doing is to write some hooks for putting some restrictions and sanitizing the code moving into the different git repos. The trick to enable hooks in gitlab is to modify the "gitlab-shell/hooks/update" file.

On an ubuntu installation, the file can be found in the following folder

/opt/gitlab/embedded/service/gitlab-shell/hooks

But, if you have followed the installation notes - https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md as in the documents, the update file is found in

/home/git/gitlab-shell/hooks

The "update" file is written in ruby. In order to abort a push, we need to exit with a "1" status. Let us create a very simple hook to disable all deletes. Add the following lines to the "update" after the line "require_relative"

require_relative '../lib/gitlab_update'

oldhash = ARGV[1]
newhash = ARGV[2]

cmd = "git cat-file -t #{newhash}"
action = IO.popen(cmd).read

if action == "delete"
    puts "deletes are not allowed"
    exit 1
end

For each push, gitlab's update script receives the old commit hash and the new commit hash. using the "git cat-file" command we are getting the action / method performed. And then we are comparing it with the "delete" string to identify if it is a delete. We are exiting with a "1" to disable a push.

We can also write another hook to disable forced push. But it is easier to modify the "gitlab-shell/lib/gitlab_update.rb" to disable forced push. There is a function here which identifies if this is a forced push. To check forced push simply call this function inside the "exec" function. Here is the code that i added to the "exec" function to disable forced push.

 def exec
    # reset GL_ID env since we already
    # get value from it
    ENV['GL_ID'] = nil

    if forced_push?
      puts "Forced push now allowed!"
      exit 1
    end
......
end

Tuesday, April 01, 2014

Auto failover of mysql master in mysql multi-master multi-slave cluster

This post is an extension to my earlier posts about multi master replication cluster multi master replication in mysql and mysql multi master replication act II The problem I had encountered and discussed was with automatic failover. What happens when the master goes down? How can either a slave or another master be promoted to become the master? Once the settings are done on all the mysql dbs in the cluster to identify the new master, we will also have to change our app to point to the new master.

Let us look at the auto failover part from the mysql cluster point of view. The tool that we are going to use is mysql master ha. It is a tool written by http://yoshinorimatsunobu.blogspot.com/ and has been in place for quite some time now. The tool supports failover in a master-master scenario, but the master-master setup has to be an active-passive(read only) setup. One of the master mysql servers has to be in read only mode. It does not support active-active mysql master-master setup where both masters are handling writes. In order to make a mysql slave read-only, simply enter "read_only" in its my.cnf file.

The mysql version I am using here is mysql 5.6.17. I have created a mysql cluster with 4 nodes.

M1(RW)--->S1
 |
 |
M2(RO)--->S2

M1 and M2 are 2 masters running on ports 3311 & 3312. S1 is a slave running on port 3313 and replicates from master M1. S2 is another slave running on port 3314 and replicates from master M2. Remember, in mysql 5.6, an additional variable has been incorporated known as the server-uuid. This has to be different for all mysql servers in the mysql cluster. So if you are replicating by copying the data directory, simply remove the auto.cnf file, which contains the server-uuid and mysql will create a new uuid when it starts. In order to make M2 read only slave, I have to put the following parameter in its my.cnf file.

read_only

Download the mysql-master-ha tool from https://code.google.com/p/mysql-master-ha/downloads/list. I am using the following version.

mha4mysql-manager-0.55.tar.gz
mha4mysql-node-0.54.tar.gz


untar the mha4mysql-node archive first and install it.

mha4mysql-node-0.54$ perl Makefile.PL
*** Module::AutoInstall version 1.03
*** Checking for Perl dependencies...
[Core Features]
- DBI        ...loaded. (1.63)
- DBD::mysql ...loaded. (4.025)
*** Module::AutoInstall configuration finished.
Writing Makefile for mha4mysql::node
Writing MYMETA.yml and MYMETA.json

mha4mysql-node-0.54$ make
mha4mysql-node-0.54$ sudo make install


Next install the mha4mysql-manager.

mha4mysql-manager-0.55$ perl Makefile.PL
*** Module::AutoInstall version 1.03
*** Checking for Perl dependencies...
[Core Features]
- DBI                   ...loaded. (1.63)
- DBD::mysql            ...loaded. (4.025)
- Time::HiRes           ...loaded. (1.9725)
- Config::Tiny          ...loaded. (2.20)
- Log::Dispatch         ...loaded. (2.41)
- Parallel::ForkManager ...loaded. (1.06)
- MHA::NodeConst        ...loaded. (0.54)
*** Module::AutoInstall configuration finished.
Writing Makefile for mha4mysql::manager
Writing MYMETA.yml and MYMETA.json

mha4mysql-manager-0.55$ make
mha4mysql-manager-0.55$ sudo make install


Watch out for errors. Once the tool is installed, we need to create a configuration file for the cluster. Here is the configuration file - app1.cnf.

[server default]
multi_tier_slave=1
manager_workdir=/home/jayantk/log/masterha/app1
manager_log=/home/jayantk/log/masterha/app1/manager.log

[server1]
hostname=127.0.0.1
candidate_master=1
port=3311
user=root
ssh_user=jayantk
master_binlog_dir=/home/jayantk/mysql-5.6.17/data1

[server2]
hostname=127.0.0.1
candidate_master=1
port=3312
user=root
ssh_user=jayantk
master_binlog_dir=/home/jayantk/mysql-5.6.17/data2

[server3]
hostname=127.0.0.1
port=3313
user=root
ssh_user=jayantk
master_binlog_dir=/home/jayantk/mysql-5.6.17/data3

[server4]
hostname=127.0.0.1
port=3314
user=root
ssh_user=jayantk
master_binlog_dir=/home/jayantk/mysql-5.6.17/data4



Let us go through it and understand what is happening. "multi_tier_slave" is a parameter used for specifying that the mysql cluster is a multi-master and multi-slave cluster where each master can have its own slave. If the parameter is not specified, mha will give an error. Next we specify the working directory and log file for mha manager. Then we start specifying each of our servers with the host name and port. We have to specify the root username and password for each of our mysql servers. The parameters are "user" and "password". I have not specified password for my mysql servers and so do not have that parameter in my configuration file. The parameter "candidate_master" is used to prioritize a certain mysql server as a master candidate during failover. In our case, we are prioritizing server2 (M2) as the master. Finally mha needs ssh access to the machines running our mysql servers. I have specified the ssh username to be used for ssh as "ssh_user". And have enabled ssh public key authentication without pass phrase for the machines (127.0.0.1 in my case). The parameter "master_binlog_dir" is used if the dead master mysql is reachable via ssh to copy any pending binary log events. This is required because there is no information with slave about where the binary log files are located.

Another thing to remember is to grant "replication slave" to all other mysql servers on the network from each mysql server in the cluster - irrespective of whether it is a master or a slave. In my case, I had to run the following grant statement on all my mysql servers - M1, M2, S1 and S2.

grant replication slave on *.* to  slave_user@localhost identified by 'slave_pwd';

Once all the settings are in place, run the command

masterha_check_repl --conf=app1.cnf

It will output a lot of messages ending with

"MySQL Replication Health is OK."


Which means that the configuration is a success. In case of errors, check the wiki at https://code.google.com/p/mysql-master-ha/wiki/TableOfContents for solutions and missing parameters.

To start the masterha manager, run the following command

masterha_manager --conf=app1.cnf

In order to check the status of the mha manager script, go to the log directory - in our case - /home/jayantk/log/masterha/app1. It has a file specifying the health status. You can cat the file to see the health status.

~/log/masterha/app1$ cat app1.master_status.health
9100    0:PING_OK    master:127.0.0.1


Another file is the log file - manager.log - which will give the following output as tail.

Tue Apr  1 12:20:50 2014 - [warning] master_ip_failover_script is not defined.
Tue Apr  1 12:20:50 2014 - [warning] shutdown_script is not defined.
Tue Apr  1 12:20:50 2014 - [info] Set master ping interval 3 seconds.
Tue Apr  1 12:20:50 2014 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Tue Apr  1 12:20:50 2014 - [info] Starting ping health check on 127.0.0.1(127.0.0.1:3311)..
Tue Apr  1 12:20:50 2014 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..


The mha manager will keep on pinging the master every 3 seconds to see if it is alive or not. The ping interval time can be changed by specifying the parameter "ping_interval" in the application configuration. Failover is triggered after missing 3 pings to the master. So if ping_interval is 3 seconds, the maximum time to discover that the master mysql is down is 12 seconds.

Now, lets kill the master and see what happens. Find out the pid of the mysqld process and kill it. It is automatically restarted by the parent process "mysqld_safe". Here is what we get in out manager.log file.

140401 12:34:12 mysqld_safe Number of processes running now: 0
140401 12:34:12 mysqld_safe mysqld restarted
Tue Apr  1 12:34:14 2014 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Tue Apr  1 12:34:14 2014 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/jayantk/mysql-5.6.17/data1 --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --binlog_prefix=zdev-bin
Tue Apr  1 12:34:15 2014 - [info] HealthCheck: SSH to 127.0.0.1 is reachable.
Tue Apr  1 12:34:17 2014 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..


It tires to do an ssh and check if the machine is reachable. But in the meantime the machine is back up an ping is available, so nothing happens. As a next step, lets kill both parent "mysqld_safe" and "mysqld" processes associated with the master mysql server in that sequence. Let us see what happens in the manager.log file. There are a lot of log entires here.


Tue Apr  1 12:44:23 2014 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
...
Tue Apr  1 12:44:26 2014 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '127.0.0.1' (111))
Tue Apr  1 12:44:26 2014 - [warning] Connection failed 1 time(s)..
Tue Apr  1 12:44:29 2014 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '127.0.0.1' (111))
Tue Apr  1 12:44:29 2014 - [warning] Connection failed 2 time(s)..
Tue Apr  1 12:44:32 2014 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '127.0.0.1' (111))
Tue Apr  1 12:44:32 2014 - [warning] Connection failed 3 time(s)..
Tue Apr  1 12:44:32 2014 - [warning] Master is not reachable from health checker!
...
Tue Apr  1 12:44:32 2014 - [info] Multi-master configuration is detected. Current primary(writable) master is 127.0.0.1(127.0.0.1:3311)
Tue Apr  1 12:44:32 2014 - [info] Master configurations are as below:
Master 127.0.0.1(127.0.0.1:3312), replicating from localhost(127.0.0.1:3311), read-only
Master 127.0.0.1(127.0.0.1:3311), dead
...
...
Tue Apr  1 12:44:32 2014 - [info] * Phase 1: Configuration Check Phase..
Tue Apr  1 12:44:32 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Multi-master configuration is detected. Current primary(writable) master is 127.0.0.1(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info] Master configurations are as below:
Master 127.0.0.1(127.0.0.1:3311), dead
Master 127.0.0.1(127.0.0.1:3312), replicating from localhost(127.0.0.1:3311), read-only

Tue Apr  1 12:44:33 2014 - [info] Dead Servers:
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info] Checking master reachability via mysql(double check)..
Tue Apr  1 12:44:33 2014 - [info]  ok.
Tue Apr  1 12:44:33 2014 - [info] Alive Servers:
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3312)
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3313)
Tue Apr  1 12:44:33 2014 - [info] Alive Slaves:
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3312)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3313)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info] Unmanaged Servers:
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3314)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3312)
Tue Apr  1 12:44:33 2014 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Forcing shutdown so that applications never connect to the current master..
...
Tue Apr  1 12:44:33 2014 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 3: Master Recovery Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] The latest binary log file/position on all slaves is zdev-bin.000009:120
Tue Apr  1 12:44:33 2014 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3312)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3313)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info] The oldest binary log file/position on all slaves is zdev-bin.000009:120
Tue Apr  1 12:44:33 2014 - [info] Oldest slaves:
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3312)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3313)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Fetching dead master's binary logs..
...
Tue Apr  1 12:44:33 2014 - [info] Additional events were not found from the orig master. No need to save.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 3.3: Determining New Master Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Apr  1 12:44:33 2014 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Apr  1 12:44:33 2014 - [info] Searching new master from slaves..
Tue Apr  1 12:44:33 2014 - [info]  Candidate masters from the configuration file:
Tue Apr  1 12:44:33 2014 - [info]   127.0.0.1(127.0.0.1:3312)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Apr  1 12:44:33 2014 - [info]     Replicating from localhost(127.0.0.1:3311)
Tue Apr  1 12:44:33 2014 - [info]     Primary candidate for the new Master (candidate_master is set)
...
Tue Apr  1 12:44:33 2014 - [info] New master is 127.0.0.1(127.0.0.1:3312)
Tue Apr  1 12:44:33 2014 - [info] Starting master failover..
Tue Apr  1 12:44:33 2014 - [info]
From:
127.0.0.1 (current master)
 +--127.0.0.1
 +--127.0.0.1

To:
127.0.0.1 (new master)
 +--127.0.0.1
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Apr  1 12:44:33 2014 - [info] Starting recovery on 127.0.0.1(127.0.0.1:3312)..
Tue Apr  1 12:44:33 2014 - [info]  This server has all relay logs. Waiting all logs to be applied..
Tue Apr  1 12:44:33 2014 - [info]   done.
Tue Apr  1 12:44:33 2014 - [info]  All relay logs were successfully applied.
Tue Apr  1 12:44:33 2014 - [info] Getting new master's binlog name and position..
Tue Apr  1 12:44:33 2014 - [info]  zdev-bin.000009:797
Tue Apr  1 12:44:33 2014 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=3312, MASTER_LOG_FILE='zdev-bin.000009', MASTER_LOG_POS=797, MASTER_USER='slave_user', MASTER_PASSWORD='xxx';
Tue Apr  1 12:44:33 2014 - [warning] master_ip_failover_script is not set. Skipping taking over new master ip address.
Tue Apr  1 12:44:33 2014 - [info] Setting read_only=0 on 127.0.0.1(127.0.0.1:3312)..
Tue Apr  1 12:44:33 2014 - [info]  ok.
Tue Apr  1 12:44:33 2014 - [info] ** Finished master recovery successfully.
Tue Apr  1 12:44:33 2014 - [info] * Phase 3: Master Recovery Phase completed.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 4: Slaves Recovery Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] -- Slave diff file generation on host 127.0.0.1(127.0.0.1:3313) started, pid: 10586. Check tmp log /home/jayantk/log/masterha/app1/127.0.0.1_3313_20140401124432.log if it takes time..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Log messages from 127.0.0.1 ...
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Tue Apr  1 12:44:33 2014 - [info] End of log messages from 127.0.0.1.
Tue Apr  1 12:44:33 2014 - [info] -- 127.0.0.1(127.0.0.1:3313) has the latest relay log events.
Tue Apr  1 12:44:33 2014 - [info] Generating relay diff files from the latest slave succeeded.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] -- Slave recovery on host 127.0.0.1(127.0.0.1:3313) started, pid: 10588. Check tmp log /home/jayantk/log/masterha/app1/127.0.0.1_3313_20140401124432.log if it takes time..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Log messages from 127.0.0.1 ...
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Starting recovery on 127.0.0.1(127.0.0.1:3313)..
Tue Apr  1 12:44:33 2014 - [info]  This server has all relay logs. Waiting all logs to be applied..
Tue Apr  1 12:44:33 2014 - [info]   done.
Tue Apr  1 12:44:33 2014 - [info]  All relay logs were successfully applied.
Tue Apr  1 12:44:33 2014 - [info]  Resetting slave 127.0.0.1(127.0.0.1:3313) and starting replication from the new master 127.0.0.1(127.0.0.1:3312)..
Tue Apr  1 12:44:33 2014 - [info]  Executed CHANGE MASTER.
Tue Apr  1 12:44:33 2014 - [info]  Slave started.
Tue Apr  1 12:44:33 2014 - [info] End of log messages from 127.0.0.1.
Tue Apr  1 12:44:33 2014 - [info] -- Slave recovery on host 127.0.0.1(127.0.0.1:3313) succeeded.
Tue Apr  1 12:44:33 2014 - [info] All new slave servers recovered successfully.
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] * Phase 5: New master cleanup phase..
Tue Apr  1 12:44:33 2014 - [info]
Tue Apr  1 12:44:33 2014 - [info] Resetting slave info on the new master..
Tue Apr  1 12:44:33 2014 - [info]  127.0.0.1: Resetting slave info succeeded.
Tue Apr  1 12:44:33 2014 - [info] Master failover to 127.0.0.1(127.0.0.1:3312) completed successfully.
Tue Apr  1 12:44:33 2014 - [info]

----- Failover Report -----

app1: MySQL Master failover 127.0.0.1 to 127.0.0.1 succeeded

Master 127.0.0.1 is down!

Check MHA Manager logs at zdev.net:/home/jayantk/log/masterha/app1/manager.log for details.

Started automated(non-interactive) failover.
The latest slave 127.0.0.1(127.0.0.1:3312) has all relay logs for recovery.
Selected 127.0.0.1 as a new master.
127.0.0.1: OK: Applying all logs succeeded.
127.0.0.1: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
127.0.0.1: OK: Applying all logs succeeded. Slave started, replicating from 127.0.0.1.
127.0.0.1: Resetting slave info succeeded.
Master failover to 127.0.0.1(127.0.0.1:3312) completed successfully.



So eventually what has happened is that the following mysql cluster is now in place

M2(RW) ---> S2
   |
   |
  V
  S1

We can log into each of the mysql servers and verify the same. So, the failover happened. There are a few problems with this solution.

  1. Now that M2 is the master, we will have to change our application to make M2 as master instead of M1. So all inserts start happening on M2. An easier way to do this is to use an HA solution known as Pacemaker which will take over the virtual ip of the master M1 and assign it to the new master M2. Another way is to change the script on all app servers and swap the ip of M1 with that of M2. This is handled by the "master_ip_failover_script" which can be configured to handle either scenario. More details on configuring the same is available here - https://code.google.com/p/mysql-master-ha/wiki/Parameters#master_ip_failover_script 
  2. The purpose of Master-Master mysql replication is lost when failover happens. Master-Master replication allows the possibility of easy switching of the app between two masters. It is even possible to write to certain databases on M1 and other databases on M2. But with this solution, firstly M2 has to be read-only, which is only acting as a slave. Secondly, after the failover, M2 loses all information which makes it a slave of M1. So, after M1 comes back up, it cannot be put into circular replication as before.
  3. The script does not handle slave failover. So it is expected that each master will have multiple slaves and the failover of slaves should be handled separately. If a mysql slave server goes down, the application should be able to identify the same and not use that particular slave server for queries. Also if mha is monitoring the mysql servers and one of the slaves goes down (undetected), and then the master mysql fails, it may not be able to make the failover.
  4. And finally, after the failover happens the script simply exits. It does not update the mha configuration file. So, multiple failovers cannot be handled. In order to start it again, we will have to manually change the configuration file and start the mha manager script again.
But, inspite of the limitations of the script, I believe it can be used in majority of the cases. And provides a good solution to mysql master failover.

Wednesday, March 19, 2014

An interesting suggester in Solr

Auto suggestion has evolved over time. Lucene has a number of implementations for type ahead suggestions. Existing suggesters generally find suggestions whose whole prefix matches the current user input. Recently a new AnalyzingInfixSuggester has been developed which finds matches of tokens anywhere in the user input and in the suggestion. Here is how the implementation looks.




Let us see how to implement this in Solr. Firstly, we will need solr 4.7 which has the capability to utilize the Lucene suggester module. For more details check out  SOLR-5378 and  SOLR-5528. To implement this, look at the searchComponent named "suggest" in solrconfig.xml. Make the following changes.

<searchComponent name="suggest" class="solr.SuggestComponent">
      <lst name="suggester">
      <str name="name">mySuggester</str>
      <str name="lookupImpl">AnalyzingInfixLookupFactory</str>
      <str name="suggestAnalyzerFieldType">text_ws</str>
      <str name="dictionaryImpl">DocumentDictionaryFactory</str>     <!-- org.apache.solr.spelling.suggest.HighFrequencyDictionaryFactory -->
      <str name="field">cat</str>
      <str name="weightField">price</str>
      <str name="buildOnCommit">true</str>
    </lst>
</searchComponent>

Here we have changed the type of lookup to use - lookupImpl to AnalyzingInfixLookupFactory. And defined the Analyzer to use for building the dictionary as text_ws - which is a simple WhiteSpaceTokenizer factory implementation. The field to be used for providing suggestions is "cat" and we use the "price" field as weight for sorting the suggestions.

Also change the default dictionary for suggester to "mySuggester". Add the following line to "requestHandler" by the name of "/suggest".

<str name="suggest.dictionary">mySuggester</str>

Once these configurations are in place, simply restart the solr server. In order to check the suggester, index all the documents in the exampleDocs folder. The suggester index is created when the documents are committed to the index. In order to check the implementation simply use the following URL.

http://localhost:8983/solr/suggest/?suggest=true&suggest.q=and&suggest.count=2

The output will be somewhat similer to this...


<response>
  <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">3</int>
  </lst>
  <lst name="suggest">
    <lst name="mySuggester">
      <lst name="and">
        <int name="numFound">2</int>
        <arr name="suggestions">
        <lst>
          <str name="term">electronics <b>and</b> computer1</str>
          <long name="weight">2199</long>
          <str name="payload"/>
        </lst>
        <lst>
          <str name="term">electronics <b>and</b> stuff2</str>
          <long name="weight">279</long>
          <str name="payload"/>
        </lst>
        </arr>
      </lst>
    </lst>
  </lst>
</response>

We are already getting the suggestions as highlighted. Try getting suggestions for some other partial words like "elec", "comp" and see the output. Let us note down some limitations that I came across while implementing this.

Checkout the type of the field "cat" which is being used for providing suggestions in schema.xml. It is of the type "string" and is both indexed and stored. We can change the field name in our solrconfig.xml to provide suggestions based on some other field, but the field has to be both indexed and stored. We would not want to tokenize the field as it may mess up with the suggestions - so it is recommended to use "string" fields for providing suggestions.

Another flaw, that I came across is that the suggestions do not work on multiValued fields. "cat" for example is multivalued. Do a search on all "electronics" and get the "cat" field.

http://localhost:8983/solr/collection1/select/?q=electronics&fl=cat

We can see that in addition to "electronics", the cat field also contains "connector", "hard drive" and "memory". But a search on those strings does not give any suggestions.

http://localhost:8983/solr/suggest/?suggest=true&suggest.q=hard&suggest.count=2

So, it is recommended that the field be of type "string" and not multivalued. If there are multiple fields on which suggestions are to be provided, it is recommended to merge them into a single "string" field in our index.

Monday, February 24, 2014

Win Free e-copies of Apache Solr PHP Integration

Readers would be pleased to know that I have teamed up with Packt Publishing to organize a giveaway of Apache Solr PHP Integration

And 3 lucky winners stand a chance to win e-copies of their new book. Keep reading to find out how you can be one of the Lucky Winners.



Overview

• Understand the tools that can be used to communicate between PHP and Solr, and how they work internally
• Explore the essential search functions of Solr such as sorting, boosting, faceting, and highlighting using your PHP code
• Take a look at some advanced features of Solr such as spell checking, grouping, and auto complete with implementations using PHP code

How to enter ?

All you need to do is head on over to the book page and look through the product description of the book and drop a line via the comments below this post to let us know what interests you the most about this book. It’s that simple.

Deadline

The contest will close on 5th March 2014 Winners will be contacted by email, so be sure to use your real email address when you comment!

Monday, December 23, 2013

A book every php developer should read

Once upon a time, a long long time ago, when there was no Solr and lucene used to be a search engine api available for php developers to use, they used to struggle for using lucene. Most people reverted back to Mysql full text search. And some ventured into using sphinx - another free full text search engine. Then came Solr and php developers were thrilled with the ease of use over the http interface for both indexing and searching text using lucene abstracted by Solr.

Even in those days, it was difficult to fully explore and use the features provided by lucene and Solr through php. There is an extension in Php to communicate to Solr. But the extension has not been in active development. As Solr came out with more and more features, the extension became very basic. Most of the advanced features provided by Solr were not available in the php Solr extension. Then came Solarium, an open source library which is being very actively developed and had support for the latest features of Solr.

But as the features of Solarium and Solr kept on increasing, php developers find it difficult to keep up to date with them. The book Apache Php Solr Integration provides an up to date and in depth view of the latest features provided by Solr and how it can be explored in php via Solarium. Php developers are generally very comfortable in writing code and in setting up systems. But in case you are a developer and are not very familiar with how to Setup solr or how to connect to Solr using php, the book hand holds you with configurations, examples and screen shots.

In addition to discussing simple topics like indexing and search which are very basic to Solr, the book also goes in depth on advanced queries in Solr like filter queries and faceting. The book also guides a developer on setting up Solr for highlighting hits in the results and goes into the implementation with sample codes. Other advanced functionalities discussed in the book are development and implementation of spell check in Solr and php, grouping of results, implementing the more like this feature in Php and Solr. The book also discusses distributed search - a feature used for Scaling Solr horizontally. Setting up of Master-Slave on Solr is discussed with sample configuration files. Load balancing of queries using php and Solarium is also discussed with sample code.

As a php developer, you may have some questions like

Q: Why should i read this book?
A: The book would make you an expert in search using Solr. That would be an additional skill that you can show off.

Q: I know Solr. What additional does the book provide ?
A: Are you up to date with the latest features provided by Solr ?  Have you implemented featues like spell check, suggestions, result grouping, more like this ?

Q: I am an expert in all the above features of Solr. What else does the book have ?
A: Are you comfortable implementing Solr on large scale sites with index which has millions of documents ? Do you know how Solr calculates relevance and how it can be tweaked ? Can you provide index statistics of Solr using php ?

If you are still undecided, the following article and table of contents of the book will help you make your mind.

http://www.packtpub.com/article/apache-solr-php-integration
http://www.packtpub.com/apache-solr-php-integration/book

Saturday, August 31, 2013

Handling mongodb connections

Mongodb is fast and for medium to large setup, it just works out of the box. But for setups which are bigger than large, you may run into a situation where the number of connections max out. So for extra large setups, let us look at how to increase the number of connections in a mongodb server.

Mongodb uses file descriptors to manage connections. In most unix-like operating systems, the default number of file descriptors available are set to 1024. This can be verified by using the command ulimit -n which should give the output as 1024. Ulimit is the per user limitation of various resources. Ulimit can be temporarily changed by issueing the command ulimit -n .

To change file descriptors system wide, change or add the following line to your /etc/sysctl.conf

fs.file-max = 64000

Now every user in your server would be able to use 64000 file descriptors instead of the earlier 1024. For a per user configuration, you will have to tweak the hard and soft limits in /etc/security/limits.conf.

By default the mongodb configuration file mostly mongodb.conf does not specify the number of max connections. It depends directly on the number of available file descriptors. But you can control it using the maxConns variable. Suppose you want to set the number of max connections to 8000, you will have to put the following configuration line in your mongodb.conf file.

maxConns = 8000

Remember : mongodb cannot use more than 20,000 connections on a server.

I recently came across a scenario where my mongodb which was using 7000 connections maxed out. I have a replica set configured, where there is a single master and multiple replicas. All reads and writes were happening on the master. With replica sets, the problem is that if the master mongo is restarted, any one of the replica mongo servers may become the master or primary. The TMC problem was caused by a missing index which caused multiple update queries to get queued up. To solve the problem, firstly the missing index was applied. Next all queries which were locking the collection had to be killed.

Here is a quick function that we were able to put together to first list all pending write operations and then kill them.

db.currentOp().inprog.forEach(
   function(d){
     if(d.waitingForLock && d.lockType != "read")
        printjson(d.opid);       
        db.killOp(d.opid);       
        i++
     })