This is a test post.

Testing A Link to Google
Testing Within the main domain
Testing Completely the same domain

Small Image:

Big Image: img_1802.jpg

Del? what now?

So, instead of working diligently on my networks assignment, I have decided to use my time to follow a different path. I am creating a new tool that will be useful for myself, and I think possibly for other people as well. My basic problem is that I’m forgetful and I need something that can remind me to do things.

My current cell phone has a nice feature of being able to add multiple alarms, which I use sometimes to help myself out. However, the alarms are clumsy to activate, don’t convey what I’m supposed to remember, and they are persistent (they stay around after my need to remember has ended). The cell phone is a good device for this though, as it is something I always have with me. It is just the interface that needs improvement.

Enter txal (’teks’əl), a web interface I’ve designed that, at a specific time, connects to my SMS message center and sends a pre-definied text message to my cell phone. It is essentially a way to send time-delayed text messages to yourself. I’m hoping that it will be easy enough that I can even convince other people to use it too, since I am assuming I’m not the only one who forgets.

I never fully believed it could happen until I saw it for myself a few minutes ago. My C++ program aborted with the following output:

  pure virtual method called
  terminate called without an active exception
  dying from signal 6: Aborted

The bizarre thing is that C++ normally checks at compile time to see if all pure virtual methods in abstract classes are implemented in subclasses (the classes you actually instantiate in your program). It would seem it is not perfect at doing this.

In actuality, the method in question was implemented. Only that after the destructor in the subclass was called, the system moved on to the base class destructor which called this method. At this point in the life cycle, it appears to have reverted back to a pure virtual method.

Thankfully in my case, switching to templates instead of inheritance to make things generic works nicely. Actually, now that I’m looking at it, I think it actually works better this way.

We have now, officially, created ContingencyWorks Inc. and are selling a product called the EpochBox. The EpochBox is a backup appliance–that is, a device that requires little to no intervention and backs up data.

We use a bunch of fancy techniques, data structures and algorithms to transmit and store the data more efficiently than any other backup appliance. Throw in RAID as a standard feature and you’ve got something worth buying! Check out a slightly better description on the product page.

So the problem was to make the following little algorithm really really fast:

• loop over many items
• for each item, do a couple of million database queries
• each query was looking for a different rowid

My initial solution was slow. Very slow.

Using SQLite, the super-fast and super-lightweight SQL server, and the standard practice of using prepared statements and persistent DB connections didn’t help at all. I was just doing an insane amount of queries and it couldn’t handle it. The creation of a DB cursor was killing me (among other things). The thing is, most of the time I was expecting it to return an empty result set–that is, I was wasting all this time creating DB cursors and empty result sets when I was mostly just testing for the existence of a record.

My first attempt at speeding this up was to create a set (as in a std::set) in memory to hold all my key values. I thought that searching a set would be much quicker since it has less overhead than a DB query. Well, unfortunately, it seems that STL sets do have a lot of overhead. I replaced DB cursors with iterators. It did speed things up slightly, but not nearly enough.

Going into more complex data structures revealed Judy Arrays, a sort of trie on steroids. Since I only needed to test for existence with it, I was able to use the Judy1 variant, which is essentially a bit-set. I was quite impressed with the space requirements, at 1,000,000 entries, each entry was using (on average) 2 bytes. Although this is larger than a standard bit-set, the Judy array lets me resize it on-the-fly which I cannot do with a standard bit-set. So, for small data-sets, it uses very little space, and for larger data-sets it scales up nicely.

So, the space is good, but what really sold me was the speed. These things were fast. The official site says it is optimized to avoid cache-line fills. Their basic rational is that if, for example, a cache-miss takes at least 50 cycles and you can avoid it with more code that runs in at most 50 cycles, you win. This does come at a cost though since the code to do this is horribly complex. At 20,000 lines, some critics say that it is too complex. My thought is that I don’t care about the code in a library so long as it works and works fast–also, when’s the last time you looked at STLs code and said “wow, that’s so simple and short?”

So, in the end, using a Judy array in my project increased speed by a few orders of magnitude. For more information about the library or the technical rational, check out the official site: http://judy.sourceforge.net/.

So, step one: write a library on Linux. Step two: port it to Windows.

Step one is quite nice, actually. We have a great build system going right now. For those familiar with make, we run at -j21. For those not familiar, this means everything builds crazy fast. Basically, we built a compile cluster of 8 dedicated machines. To this, every developer’s workstation is added. This gives us a huge amount of compiling power.

Now, on the other hand, our Windows build system sucks. It consists of one machine, and thus no distributed builds. Heck, the one machine doesn’t even have a decent processor. I believe it is an 800 MHz Celeron. Compiling on this thing is quite painful. I’ve basically decided that once we have some more resources, I’m getting a faster Windows machine.

Well, I’ve decided to reinstall the OS on develotron. Mostly, it was overdue for an update. I could just upgrade the existing, but I’ve decided to do it from scratch just to make sure everything is clean. Things that I’m going to have to set up include: LDAP based PAM authentication, apache2, trac, sugarcrm, subversion.

Upgrading Berkeley DB:
This was interesting. So, the old version of Subversion used bdb4.2 but the current version uses bdb4.3. All well and good except that their log files are incompatible. I got around this one by doing the following:

• installing db4.3-util and db4.2-util
• running db4.2_recover in svnrepo/db
• deleting my log files (log.*)

Now subversion can access the repo using the new version of Berkeley DB.

Subversion permissions:
Since there are multiple users accessing the Subversion repository concurrently, permissions for the Berkeley DB sometimes get screwy. A simple fix is as follows:

• make everything owned, readable, and writable by a group svn
• chmod g+s repo/db
• move svn to svn_real and create a shell script called svn:
  

    #!/bin/sh
    umask 0002
    /usr/bin/svn_real "$@"
    

• move svnserve to svnserve_real and create a shell script called svnserve:
  

    #!/bin/sh
    umask 0002
    /usr/bin/svnserve_real "$@"
    

That should do it! Now the database files will be created with the proper permissions and ownership so that everyone in the svn group can still access them.

Installing OpenLDAP and pam_ldap:
This was actually simpler than I remembered–once I figured out what to actually do. Obviously, the first step was to install OpenLDAP. This was as simple as typing aptitude install openldapd openldap-utils. When you do this, make sure to set the base dn and remember what you set it to; you need this numerous times when setting up pam.

Next install libpam-ldap and libss-ldap. Now all the packages you need are installed, they just aren’t active. Change your /etc/pam_ldap.conf file and /etc/libnss-ldap.conf to have at least the following:

  host                   127.0.0.1
  base                   dc=what_you_put,dc=before
  ldap_version           3
  pam_password           exop
  nss_base_passwd        ou=People,dc,what_you_put,dc=before
  nss_base_shadow        ou=People,dc=what_you_put,dc=before
  nss_base_group         ou=Groups,dc=what_you_put,dc=before
  nss_map_objectclass    shadowAccount    posixAccount

Now that PAM and libnss know how to use LDAP, let’s tell them that they should use it. Edit your /etc/nsswitch.conf like so:

  passwd:    compat ldap
  group:     compat ldap
  shadow:    compat ldap

You could add LDAP support for hosts, networks, protocols, etc. but I don’t bother.

The next thing is to set up PAM to use LDAP as well. Assuming you are on a system with a similar PAM setup to Ubuntu, the file you want to edit is /etc/pam.d/common-auth and it needs the following two lines:

  auth  sufficient  pam_ldap.so ignore_authinfo_unavail
  auth  required    pam_unix.so use_first_pass nullok_secure

This tells PAM to try LDAP first if it can, if the user exists in the LDAP directory that is good enough, otherwise, fall back and try authenticating against the standard UNIX /etc/passwd (using the same password that we tried to use with LDAP).

Update wow. I forgot I had this draft saved. It is now a few months later. I guess I will just post what is here. I don’t remember what else I did anymore.

http://sam.zoy.org/writings/programming/gprof.html

A shared library that you can LD_PRELOAD to fix profiling from within pthread. Something to do with a certain signal not being inherited after calling clone()–this library fixes that problem.

This script/stylesheet combo lets you annotate images similar to what Flickr does.

http://kryogenix.org/code/browser/annimg/annimg.html

So, our VoIP server has moved out of the back of my bedroom and into a proper data center. My room is so quiet now. I can actually hear things again. It is for the best though. Fivegrand needed a more reliable internet connection to serve its purpose. It now has that–and so much more.