Showing posts with label programming. Show all posts
Showing posts with label programming. Show all posts

Tuesday, August 05, 2008

Berkeley DB

I had a chance to look in BDB-XML some time back. But i did not find it that good in performance. BDB on the other hand promises fast storage and retrieval of key-value pairs. For the new-comers, BDB is a embedded database API which can be used to create your own database and fire a defined set of queries on it. BDB does not provide you with an SQL interface or even a database server. All you have is a set of API using which you can write programs to create your own database, populate data into the database, define your own cache and indexes and then use the database to retrieve values for a particular key using your own programs.

With BDB, there is no sql/query optimizer/query parser layer between you and the database.

There are certain types of BDB engines available:

1.] BDB -> the original BDB with c/c++ api. You write programs in c/c++ to create and access your database.
2.] BDB-JAVA -> The java api to BDB. The java api uses JNI in the backend to communicate with the BDB library (written in c/c++).
3.] BDB-JE -> The java edition of the BDB engine. It is a pure java implementation of the BDB engine. It does not use JNI for communication with the system.
4.] BDB-XML -> This is a very sophesticated version of BDB - where you can store XML documents and retrieve documents using any of the keys in the XML Document. You have an XQuery interface where you can fire XML based queries and retrieve results.

The original BDB is ofcourse the fastest.

For a startup, we will take a look at the DPL API of BDB-JAVA. DPL stands for Direct Persistence Layer and is used generally for storing and managing java objects in the database. DPL works best with a static database schema and requires java 1.5.

To create a database using DPL, you generally require an entity class and then create/open a database environment and insert the entity object into the entity store in the database environment. Sounds greek right ?? Lets see an example

Entity Class

import com.sleepycat.persist.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.model.*;
import static com.sleepycat.persist.model.Relationship.*;

@Entity
public class SimpleEntityClass {

   // Primary key is pKey
   @PrimaryKey
   private String pKey;

   // Secondary key is the sKey
   @SecondaryKey(relate=MANY_TO_ONE)
   private String sKey;

   public SimpleEntityClass(String pk, String sk)
   {
      this.pKey = pk;
      this.sKey = sk;
   }

   public void setpKey(String data)
   {
      pKey = data;
   }

   public void setsKey(String data)
   {
      sKey = data;
   }

   public String getpKey()
   {
      return pKey;
   }

   public String getsKey()
   {
      return sKey;
   }
}


Then create a Database Access class which can insert and retrieve Entity objects from the database.

import java.io.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.*;

public class SimpleDA
{
   PrimaryIndex pIdx;
   SecondaryIndex sIdx;

   EntityCursor pcursor;
   EntityCursor scursor;

   public SimpleDA(EntityStore store) throws Exception
   {
      pIdx = store.getPrimaryIndex(String.class, SimpleEntityClass.class);
      sIdx = store.getSecondaryIndex(pIdx, String.class, "sKey");
   }

   public void addEntry(String pk, String sk) throws DatabaseException
   {
      pIdx.put(new SimpleEntityClass(pk, sk));
   }

   public SimpleEntityClass findByPk(String pk) throws DatabaseException
   {
      SimpleEntityClass found = pIdx.get(pk);
      return found;
   }

   public ArrayList findBySk(String sk) throws DatabaseException
   {
      ArrayList ret = new ArrayList();
      scursor = sIdx.subIndex(sk).entities();
      for(SimpleEntityClass sec1 = scursor.first(); sec1!=null; sec1 = scursor.next())
      {
         ret.add(sec1);
      }
      scursor.close();
      return ret;
   }
}


Create/open the environment to put and retrieve records from the database

import java.io.*;
import com.sleepycat.db.*;
import com.sleepycat.persist.*;

public class SimpleStore
{
   private static File envHome = new File("./bdbjava");
   private Environment env;
   private EntityStore store;
   private SimpleDA sda;

   public void setup() throws DatabaseException
   {
   // put all config options here.
      EnvironmentConfig envConfig = new EnvironmentConfig();
      envConfig.setAllowCreate(true);
      envConfig.setCacheSize(536870912); //512 MB
      envConfig.setCacheCount(2); // 2 caches of 256 MB each
      envConfig.setTxnWriteNoSync(true);
      envConfig.setInitializeCache(true);
      envConfig.setThreaded(true);
      envConfig.setInitializeLogging(true);
      envConfig.setTransactional(true);

      StoreConfig sConfig = new StoreConfig();
      sConfig.setAllowCreate(true);

      try
      {
         env = new Environment(envHome, envConfig);
         store = new EntityStore(env, "MyDatabaseName", sConfig);
      }catch(Exception ex)
      {
         ex.printStackTrace();
         System.exit(-1);
      }
   }

   public SimpleStore()
   {
      setup();
   }

   public void putData(String pk, String sk) throws Exception
   {
      sda = new SimpleDA(store);
      sda.addEntry(pk, sk);
   }

   public void getDataPk(String pk) throws Exception
   {
      sda = new SimpleDA(store);
      SimpleEntityClass sec = sda.findByPk(pk);
      System.out.println("pk = "+sec.getpKey()+", sk = "+sec.getsKey());
   }

   public void getDataSk(String sk) throws Exception
   {
      sda = new SimpleDA(store);
      ArrayList data = sda.findBySk(sk);
      for(int x=0; x<data.size(); x++)
      {
         SimpleEntityClass sec = data.get(x);
         System.out.println("pk = "+sec.getpKey()+", sk = "+sec.getsKey());
      }
   }

   public void closeAll() throws Exception
   {
      store.close();
      env.close();
   }

   public static void main(String[] args)
   {
      SimpleStore ss = new SimpleStore();
      ss.putData("pk1","sk1");
      ss.putData("pk2","sk2");
      ss.putData("pk3","sk1");
      ss.putData("pk4","sk3");

      ss.getDataPk("pk1");

      ss.getDataSk("sk1");

      ss.closeAll();
   }
}


So, now you have your own program for creating entities of the type SimpleEntityClass and store them in the database in serialized form. These objects cab be retrieved using primary or secondary keys.

For the relationship between primary and secondary keys please refer to http://www.oracle.com/technology/documentation/berkeley-db/db/java/com/sleepycat/persist/model/Relationship.html

Since you would be storing complete objects instead of just the required data sets, the database size would be relatively high and that would slow down things a bit.

Friday, May 30, 2008

python programming - threading

There are a few options for threading in python. I wont be exploring all the options here. I would simply try to make this blog entry simple, sweet and short.

There are two modules in python which provide threading capabilities - "thread" module and "threading" module. The thread module is very basic. So lets focus on the threading module

To use the threading module, all you have to do is

import threading

class mythread(threading.Thread):
  def run(self):
    <your code>

mythread().start()
mythread().start()


This will create 2 threads running <your code>

That was quick right. Now lets see some basic stuff like Thread Naming, Thread isAlive and join which is used in most threading environments...

Naming a thread:

import threading

class mythread(threading.Thread):
  def run(self):
    print 'my name is ', self.getName()

foo = mythread()
foo.setName('Foo')
foo.start()

bar = mythread()
bar.setName('Bar')
bar.start()

mythread().start()


Run the program:

$ python threadname.py

And see the output

my name is Foo
my name is Bar
my name is Thread-3


Checking if the thread is still alive:

import threading
import time

class mythread(threading.Thread):
  def run(self):
    print 'my name is ', self.getName()

class aliveth(threading.Thread):
  def run(self):
    time.sleep(10)
    print 'my name is ', self.getName()

myt = mythread()
myt.setName('mythread')
myt.start()
if myt.isAlive():
  print 'myt is alive'
else:
  print 'myt is dead'

alt = aliveth()
alt.setName('aliveth')
alt.start()
if alt.isAlive():
  print 'alt is alive'
else:
  print 'alt is dead'

And check the output

my name is mythread
myt is dead
alt is alive
my name is aliveth


Joining threads:

You can use the thread.join() method to make a thread wait for another thread

import threading
import time

class ThreadOne ( threading.Thread ):
  def run ( self ):
    print 'Thread', self.getName(), 'started.'
    print self.getName(), ': sleeping '
    time.sleep ( 5 )
    print 'Thread', self.getName(), 'ended.'

class ThreadTwo ( threading.Thread ):
  def run ( self ):
    print 'Thread', self.getName(), 'started.'
    print self.getName(), ': waiting for ', thingOne.getName()
    thingOne.join()
    print 'Thread', self.getName(), 'ended.'

class ThreadThree (threading.Thread):
  def run(self):
    print 'Thread', self.getName(), 'started'
    print self.getName(),': Not waiting for any other thread'
    print 'Thread', self.getName(), 'ended.'

thingOne = ThreadOne()
thingOne.start()
thingTwo = ThreadTwo()
thingTwo.start()
thingThree = ThreadThree()
thingThree.start()


And check the output

Thread Thread-1 started.
Thread-1 : sleeping
Thread Thread-2 started.
Thread-2 : waiting for Thread-1
Thread Thread-3 started
Thread-3 : Not waiting for any other thread
Thread Thread-3 ended.
Thread Thread-1 ended.
Thread Thread-2 ended.


This covers most of the stuff for programming threads in python. We will look into thread synchronization issues some other time.

Monday, October 15, 2007

Why not rails?

SUMMARY: I spent two years trying to make Rails do something it wasn’t meant to do, then realized my old abandoned language (PHP, in my case) would do just fine if approached with my new Rails-gained wisdom.

INTRO / BACKGROUND:

Back in January 2005, I announced on the O’Reilly blog that I was going to completely scrap over 100,000 lines of messy PHP code in my existing CD Baby (cdbaby.com) website, and rewrite the entire thing in Rails, from scratch.

I hired one of the best Rails programmers in the world (Jeremy Kemper aka bitsweat), and we set off on this huge task with intensity. The first few months showed good progress, and Jeremy could not have been more amazing, twisting the deep inner guts of Rails to make it do things it was never intended to do.

But at every step, it seemed our needs clashed with Rails’ preferences. (Like trying to turn a train into a boat. It’s do-able with a lot of glue. But it’s damn hard. And certainly makes you ask why you’re really doing this.)

Two years (!) later, after various setbacks, we were less than halfway done.* (To be fair to Jeremy’s mad skillz: many setbacks were because of tech emergencies that pulled our attention to other internal projects that were not the rewrite itself.) The entire music distribution world had changed, and we were still working on the same goddamn rewrite. I said fuckit, and we abandoned the Rails rewrite. Jeremy took a job with 37 Signals, and that was that.

I didn’t abandon the rewrite IDEA, though. I just asked myself one important question:

“Is there anything Rails can do, that PHP CAN’T do?”

The answer is no.

I threw away 2 years of Rails code, and opened a new empty Subversion respository.

Then in a mere TWO MONTHS, by myself, not even telling anyone I was doing this, using nothing but vi, and no frameworks, I rewrote CD Baby from scratch in PHP. Done! Launched! And it works amazingly well.

It’s the most beautiful PHP I’ve ever written, all wonderfully MVC and DRY, and and I owe it all to Rails.

Inspired by Rails:

*- all logic is coming from the models, one per database table, like Martin Fowler’s Active Record pattern.

*- no requires or includes needed, thanks to __autoload.

*- real MVC separation: controllers have no HTML or business-logic, and only use REST-approved HTTP. (GET is only get. Any destructive actions require POST.)

*- all HTML coming from a cute and powerful templating system I whipped up in 80 lines, all multi-lingual and caching and everything

*- … and much more. In only 12,000 lines of code, including HTML templates. (Down from 90,000, before.)

Though I’m not saying other people should do what I’ve done, I thought I should share my reasons and lessons-learned, here:

SEVEN REASONS I SWITCHED BACK TO PHP AFTER 2 YEARS ON RAILS:

#1 - “IS THERE ANYTHING RAILS/RUBY CAN DO THAT PHP CAN’T DO? … (thinking)… NO.”
For 2 years, I thought Rails is genius, PHP is shit. Rails is powerful, PHP is crap.
I was nearly killing my company in the name of blindly insisting Rails was the answer to all questions, timeframes be damned.
But when I took a real emotionless non-prejudiced look at it, I realized the language didn’t matter that much.
Ruby is prettier. Rails has nice shortcuts. But no big shortcuts I can’t code-up myself in a day if needed.
Looked at from a real practical point of view, I could do anything in PHP, and there were many business reasons to do so.

#2 - OUR ENTIRE COMPANY’S STUFF WAS IN PHP: DON’T UNDERESTIMATE INTEGRATION
By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members’ Login Area, and dozens of cron shell scripts were ALL going to have to change. 85 employees re-trained. All customers and clients calling up furious that One Big Day, with questions about the new system.
Instead, I was able to slowly gut the ugly PHP and replace it with beautiful PHP. Launch in stages. No big re-training.

#3 - DON’T WANT WHAT I DON’T NEED
I admire the hell out of the Rails core gang that actually understand every line inside Rails itself. But I don’t. And I’m sure I will never use 90% of it.
With my little self-made system, every line is only what’s absolutely necessary. That makes me extremely happy and comfortable.

#4 - IT’S SMALL AND FAST
One little 2U LAMP server is serving up a ton of cdbaby.com traffic damn fast with hardly any load.

#5 - IT’S BUILT TO MY TASTES
I don’t need to adapt my ways to Rails. I tell PHP exactly what I want to do, the way I want to do it, and it doesn’t complain.
I was having to hack-up Rails with all kinds of plugins and mods to get it to be the multi-lingual integration to our existing 95-table database.
My new code was made just for me. The most efficient possible code to work with our exact needs.

#6 - I LOVE SQL
Speaking of tastes: tiny but important thing : I love SQL. I dream in queries. I think in tables.
I was always fighting against Rails and its migrations hiding my beloved SQL from me.

#7 - PROGRAMMING LANGUAGES ARE LIKE GIRLFRIENDS: THE NEW ONE IS BETTER BECAUSE *YOU* ARE BETTER
Rails was an amazing teacher. I loved it’s “do exactly as I say” paint-by-numbers framework that taught me some great guidelines.
I love Ruby for making me really understand OOP. God, Ruby is so beautiful. I love you, Ruby.
But the main reason that any programmer learning any new language thinks the new language is SO much better than the old one is because he’s a better programmer now! You look back at your old ugly PHP code, compared to your new beautiful Ruby code, and think, “God that PHP is ugly!” But don’t forget you wrote that PHP years ago and are unfairly discriminating against it now.
It’s not the language (entirely). It’s you, dude. You’re better now. Give yourself some credit.

Ok. All that being said, I’m looking forward to using Rails some day when I start a brand new project from scratch, with Rails in mind from the beginning.

But I hope that this reaches someone somewhere thinking, “God our old code is ugly. If we only threw it all away and did it all over in Rails, it’d be so much easier!”

Copied from the original post http://www.oreillynet.com/ruby/blog/2007/09/7_reasons_i_switched_back_to_p_1.html

Friday, May 04, 2007

ctrl C & ctrl V

Your Colleague: Hey!! Kya yahan baitha mail forward karta rahta hai yaar!! Naye packages dekh.... Naye language seekh, Night out Maar....Fundoo programming kar like me....! Do something cool man!!

You: Achha! To usse Kya hoga...

You 're Colleague: Impression!! ! Appraisal!!! Har appraisal main tu No 1! Hike in salary!! Extra Stocks

You: Phir kya hoga...

Your Colleague: Project Leader ban jaayega..Phir Project Manager!!! Phir Business Manager! One day U will be a Director of the Company man !!

You: Acchha to phir kya hoga...

Your Colleague: Abe phir tu aish karega! Koi kaam nahin karna padega! Araam se office aayega aur MAIL check karega.

You: To ab main kya kar raha hoon????


"Dikhawe pe na jao, apni akal lagao.

Programming hai waste, trust only copy-paste "


Powered by ctrl C


Driven by ctrl V

Thursday, September 28, 2006

python

Yes python. No, I am not refering to the snake, I am refering to the programming language "python". OOPS!!! Sorry for the confusion.

Python is an interpreted, interactive and object oriented programming language. Often compared to perl, java, ruby etc. As per http://en.wikipedia.org/wiki/Python_programming_language, python was created by some dude named "Guido van Rossum" in 1990. It uses automatic memory management and is a High level language. Oooh.. And python is hot... Why?? Cause it has a very clean syntax and is easy to use and packs a lot of power inside it... Thats why, i think so... So lets get some tits and bits about python...

But before i start, just check out http://www.python.org, the official website of python software foundation the creators and maintainers of python.

The thing which will strike you at the start is the way indentation is important in python. Python groups sentenses on the basis of how it is indented. So i would write a for loop as follows


>>> arr = ['a1','b2','c3','d4','e5','f6'] # defining the array
>>> for x in arr: # starting the for loop
>>> print x # print the element and its length
>>> print len(x)
>>> print 'end' # print end the for loop. Indentation says for loop is over


The best thing i like about this is that developers would automatically indent their code making it good to look at - ummm beautiful may be the right word.

python has a nice set of string operators and regular expressions. You can create and iterate over a range very nicely, write functions, create stacks and queues, do file handling and exception handling. You have complete java style - try->catch level exception handling. And ofcourse you can create classes. Serialize and store objects in file very easily and then restore the objects later. Also there are lots of extensions available which would allow you to interact with the net, create threads, do data compression and decompression. control garbage collection.

Wooo, and the main thing is that all this is possible with very easy and clean syntax . I have seen small and large - in fact very large programs being written in python. And maybe sometime down in future python would replace perl for small programs.

You have got to try it out to see the power of python...

Saturday, February 25, 2006

C Program to Propose a girl -> too gud

/*C Program to Propose a girl*/

#include "STD_ISD_PCO.h"
#include "love.h"
#define Cute beautiful_lady
main()
{
goto college;
scanf("100%",&ladies);
if(lady ==Cute )
line++;
while( !reply ){
printf("I Love U");
scanf("100%",&reply);
}
if(reply == "GAALI")
main(); /* go back and repeat the process */
else if(reply == "SANDAL ")
exit(1);
else if(reply == "I Love U"){
lover =Cute ;
love = (heart*)malloc(sizeof(lover));
}
goto restaurant;
restaurant:{
food++;
smile++;
pay->money = lover->money;
return(college);
}
if(time==2.30)
goto cinema;
cinema:
{
watch++;
if(intermission){
coke++;
Popecorn++;}
}
if(time ==6.00)
goto park;
park:
{
for(time=6.30;time<= 8.30;time+=0.001)
kiss = kiss+1;
}
free(lover);
return(home);
if(time ==9.30)
goto pub;
pub:{
friends++;
party++;
booze++;
smoke++;
if(pub.close()){ pay->bill;
come->out;}
}
if (highly->intoxicated)
goto friendsroom;
else{
sweetpan++;
polo++;
}
friendsroom: { goto sleep; }
sleep: { *(dream)=love; }