Ryan Day

Nothing interesting has been happening lately. A few side projects that are neat, but other then that no new coding advancements. I was playing with memoization a bit. Of course, in my everyday job I have absolutely zero need for this. But maybe I will find something useful. I think that I could try to implement some advanced code techniques at work, but we would have to do things better first. Hundreds of different pages all using sql queries, no real structure. It gets mind bottling after awhile.

But anyways. To explore memoization, lets use my favorite advanced math algorithm interpreter, PHP. Then we will experiment in something new to me, C#!!!!

The fibonacci:

 PHP | 
 copy code |
?
 
1
function fib($c) {
2
   if( $c == 0 ) return 0;
3
   if( $c == 1 ) return 1;
4
   return fib($c-2) + fib($c-1);
5
}

Ok, so we have a nice recursive function to calculate the nth number in the fib sequence. Lets use it to calculate the 20th, 30th, and 40th numbers.

Start
6765: Time: 0.012
832040: Time: 1.178
63245986: Time: 135.191

Whoa, so times really start to add up quick. But we have to calculate this number or we get fired(this makes as much sense as anything else I do so I’m going to stick with it) and we have to calculate it quickly! We can use memoization on this fib() function to speed up the calls. This means that we will cache results as we calculate them, and then if we try to calculate the same number, we will grab the result from cache instead of running all the calculations again. This works because the results for fib(x) are always the same. There is no user input, time values, or any other real world nonsense getting in the way. We are going to add a static array to the fib() function, and store/retrieve the results from that array. Here is the code:

 PHP | 
 copy code |
?
 
1
function fib($c) {
2
   static $map;
3
   if( !is_array($map) ) $map = array();
4
   if( $c == 0 ) return 0;
5
   if( $c == 1 ) return 1;
6
   if( !isset($map[$c]) )        // Have we already calculated this?
7
   	$map[$c] = fib($c-2) + fib($c-1);
8
   return $map[$c];
9
}

We have a check to see if the request value has already been calculated. If not, we calculate it and store it. Then we return the stored version. This should speed things up. Lets run it again:

Start
6765: 0.001
832040: 0.002
102334155: 0.002

That is a serious improvement. It should be, we are only making 40 calls instead of the previous huge number. Now I just need to find a situation where this may be useful in my day to day programming.

What about a new language, C#? Lets run the same code again:

 C# | 
 copy code |
?
 
01
        static int fib(int x)
02
        {
03
            if (x < 2) return x;
04
            return fib(x - 2) + fib(x - 1);
05
        }
06
 
07
        static void Main(string[] args)
08
        {
09
            DateTime startTime = DateTime.Now;
10
            DateTime endTime;
11
            Console.WriteLine("Place 20: {0}", fib(20));
12
            endTime = DateTime.Now;
13
            Console.WriteLine("Time: {0}", startTime - endTime);
14
            Console.WriteLine("Place 30: {0}", fib(30));
15
            endTime = DateTime.Now;
16
            Console.WriteLine("Time: {0}", startTime - endTime);
17
            Console.WriteLine("Place 40: {0}", fib(40));
18
            endTime = DateTime.Now;
19
            Console.WriteLine("Time: {0}", startTime - endTime);
20
 
21
            Console.ReadKey();
22
        }

Note: I’m posting all the code on the off chance somebody reads my blog and would like bolster their self image by destroying my skills here. Look at the times though:

Place 20: 6765
Time: 00:00:00
Place 30: 832040
Time: -00:00:00.0901296
Place 40: 102334155
Time: -00:00:09.8040976

I mean, come on, 9 seconds as compared to PHP’s 135 seconds!? I’m running 2 virtual machines on my host machine with only 2 gigs of ram, listening to a podcast, and c# runs under windows of all things. Reality is shattering before my eyes.

Anyways. The goal here is to implement the map in C#. Obviously C# will scoff at trying to use key=>value arrays. I’ve come across a few different way to do this, and it looks like a Hashtable is the simplest.

 C# | 
 copy code |
?
 
1
        static Hashtable map;
2
 
3
        static int fib(int x)
4
        {
5
            if (x < 2) return x;
6
            if(!map.ContainsKey(x))
7
                map.Add(x, fib(x - 2) + fib(x - 1));
8
            return (int)map[x];
9
        }

Our times?

Place 20: 6765
Time: -00:00:00.0100144
Place 30: 832040
Time: -00:00:00.0100144
Place 40: 102334155
Time: -00:00:00.0100144

Not bad. There are some other ways to do this without the hashtable. You can use a ListCollection for example. But this seems like it works. Now to learn the more indepth aspects of hashtables in c#.

I needed to add Excel capability to some of my reports in Recess! so I decided a plugin would be the best way to go. There isn’t too much documentation on plugins so far, so I hope this can help a little. This may not be the best way to go, but it can at least be a guide to get started. I chose PHPExcel to be the engine I use to generate the Excel files. Its easy to generate several different versions of files, and csv and pdf, and I like the library.

The entire plugin is downloaded here.

The Recess! way to do this is create a View that Recess can use to display data. The views are called from the controller, so I want to use the view as follows:

 PHP | 
 copy code |
?
 
1
/*
2
 * !RespondsWith Layouts, ExcelView
3
 */

The first thing I had to do was create the plugin directory structure and copy the PHPExcel files in. I copied the PHPExcel Classes/ directory into plugins/ryanday/PHPExcel/. Because the Library::import() method assumes FILE.class.php, I had to link PHPExcel.class.php to the actual PHPExcel.php file for things to function smoothly. There may be a way around that, but I figure its part of Recess!izing the library. This is probably a little confusing, so this is what the directory looks like:

• Classes/
• ExcelView.class.php
• PHPExcelPlugin.class.php

Classes/
• PHPExcel/
• PHPExcel.class.php -> PHPExcel.php
• PHPExcel.php

Please download a copy of the plugin to make it a little more clear.

I started by viewing thebeeline’s post about plugins to get a jumping off point. Then I created the plugin file, PHPExcelPlugin.class.php

 PHP | 
 copy code |
?
 
01
<?php
02
Library::import('recess.framework.Plugin');
03
Library::import('ryanday.PHPExcel.Classes.PHPExcel');
04
Library::import('ryanday.PHPExcel.ExcelView');    // This will let us create our own view
05
 
06
class PHPExcelPlugin extends Plugin {
07
 
08
        function init(Application $app) {
09
                MimeTypes::register('xls','application/vnd.ms-excel');
10
        }
11
 
12
};
13

In the init() function I register a new MimeType, ‘xls’. When Recess! can’t find a layout to display data, it will pass registered mimetypes to all the views, so we have to register the ‘xls’ mimetype so our plugin can respond. Recess! will automatically create the headers for xls, and the applications type. Next we create the actual view.

 PHP | 
 copy code |
?
 
01
<?php
02
Library::import('recess.framework.AbstractView');
03
 
04
class ExcelView extends AbstractView {
05
        //  This is were we respond to the MimeType we previously registered
06
	public function canRespondWith(Response $response) {
07
		return 'xls' === $response->request->accepts->format();
08
	}
09
 
10
        /*  Most views render the model in a specific way, a spreadsheet is a little different
11
         * though. I require the xls property to be a two dimensional array containing
12
         * the data to be displayed.
13
         */
14
	protected function render(Response $response) {
15
		$objPHPExcel = new PHPExcel();
16
		require_once PHPEXCEL_ROOT . 'PHPExcel/IOFactory.php';
17
		require_once PHPEXCEL_ROOT . 'PHPExcel/Writer/Excel5.php';
18
		$objPHPExcel->setActiveSheetIndex(0);
19
 
20
                /*
21
                 * The following is loosely based on the JsonView code. If we have a model
22
                 * with the xls property set, we create the spreadsheet from that array
23
                 */
24
		$response = clone $response;
25
		foreach($response->data as $key => $value) {
26
			if($value instanceof Model) {
27
				if( isset($value->xls) ) {
28
					foreach($value->xls as $rowId=>$row) {
29
						foreach($row as $colId=>$cell) {
30
							$objPHPExcel->getActiveSheet()->setCellValueByColumnAndRow($colId, $rowId+1, $cell);
31
						}
32
					}
33
				}
34
			}
35
			if($value instanceof Form) {
36
				unset($response->data[$key]);
37
			}
38
			if(substr($key,0,1) == '_') {
39
				unset($response->data[$key]);
40
			}
41
		}
42
 
43
		if(isset($response->data['application'])) unset ($response->data['application']);
44
		if(isset($response->data['controller'])) unset ($response->data['controller']);
45
 
46
                /* I have gone through the docs and can't get ExcelWriter to dump to stdout,
47
                 * this way is a little hack I think, but it works.
48
                 */
49
		$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'Excel5');
50
		$tempfile = tempnam('/tmp', 'recessxls');
51
		$objWriter->save($tempfile);
52
 
53
                // We don't have to mess with headers, Recess! will do that based 
54
                // on the registered Mimetype
55
		readfile($tempfile);
56
	}
57
};
58

So now all our plugins files are setup, and we have to import the plugins into our application. In your app/YourApp/YourAppApplication.class.php add the following lines

 PHP | 
 copy code |
?
 
1
Library::import('ryanday.PHPExcel.PHPExcelPlugin');   // Add this import
2
 
3
class YourAppApplication extends Application {
4
   ...
5
   $this->plugins = array(new PHPExcelPlugin());   // instantiate the class to import the library
6
   $this->plugins[0]->init($this);                    // Important!!
7
}

Note the init() call! The MimeType is verified before the plugin is initialized by the Controller. That means the MimeType won’t be registered when the controller looks for it. So we do it ourselves. Finally, in our controller, we create a little example.

 PHP | 
 copy code |
?
 
01
/**
02
 * !RespondsWith Layouts, Excel
03
 * !Prefix mainModel/
04
 */
05
class MainModelController extends Controller {
06
 
07
. . .
08
 
09
       /** !Route GET, $id */
10
        function details($id) {
11
                $this->mainModel->id = $id;
12
                if($this->mainModel->exists()) {
13
                        $this->mainModel->xls = array(array("col1","col2","col3"),
14
                                        array("newrow","newcol","keepgoing"),
15
                                        array("lastrow","k","lastcol"));
16
                        return $this->ok('details');
17
                } else {
18
                        return $this->forwardNotFound($this->urlTo('index'));
19
                }
20
        }
21

Now when you visit that particular route, you will get an Excel spreadsheet with the data set in $this->mainModel->xls.

This is just a basic intro on how to create a View plugin. Please comment with any updates, or better way to explain things! This works for me so far, but it hasn’t been tested along a wide distribution of sites.

XBee Monitor is a pretty scaled down wxWidgets program that allows you to talk with and XBee coordinator in API mode and get basic information about the network. Right now only a few things are supported:

1. Node Discovery
2. Get local address
3. RX Packet disassembly

I find it useful to help make sure my devices are actually reaching my coordinator and also to see what data the devices are sending. As I keep going with my own XBee experiments I’ll keep updating this as well. I use the ttListCtrl class to handle tooltips like I spoke about previously.

You can view the code here.

I had a basic need to show Tooltips over individual wxListItems in a wxListCtrl. wxListItems don’t let you assign tooltips, and I couldn’t get the SetData() method to store data on a per-cell basis. There also isn’t any Hover event, or MouseOver event, that we can listen for. I wrote this code to enable ToolTips for each individual cell in a wxListCtrl. First, what has to be solved:

• As you add items to the wxListCtrl, provide a method to add tooltips at the same time
• Listen for a MouseOver event on the wxListCtrl, and determine the X/Y coordinate of the mouse pointer(wxPoint)
• With the wxPoint, determine which cell the mouse is over(by row/col)
• Display a tooltip window with that cell’s tooltip
• Close that tooltip window in a graceful way(proper amount of delay)

This was a lot to do to enable nice tooltips. But I really don’t have much else going on so who cares. In order to solve step 1, I decided to extend wxListCtrl and add some new funcitonality. Here is the code for the ttListCtrl class, I’ll go over it.

 C++ | 
 copy code |
?
 
01
/* ttListCtrl.h */
02
/* Ryan Day, http://www.ryanday.net/ */
03
 
04
#include <wx/listctrl.h>
05
 
06
class ttListCtrl : public wxListCtrl
07
{
08
private:
09
   wxString *grid;
10
   int cols, rows;
11
 
12
public:
13
   ttListCtrl() { };
14
   ttListCtrl( wxWindow *parent,
15
                wxWindowID winid = wxID_ANY,
16
                const wxPoint& pos = wxDefaultPosition,
17
                const wxSize& size = wxDefaultSize,
18
                long style = wxLC_REPORT,
19
                const wxValidator& validator = wxDefaultValidator,
20
                const wxString &name = wxListCtrlNameStr)
21
   {
22
       //  Start with a 3x3 grid, and we can expand beyond that if necessary
23
       cols = rows = 3;
24
       grid = new wxString[3*3];
25
       Create(parent, winid, pos, size, style, validator, name);
26
   }
27
 
28
   int SetTooltip(int row, int col, wxString& tip);
29
   int GetTooltip(int row, int col, wxString& tip);
30
};
31
 
32
/* ttListCtrl.cpp */
33
/* Ryan Day, http://www.ryanday.net/ */
34
#include "ttListCtrl.h"
35
 
36
int ttListCtrl::SetTooltip(int row, int col, wxString& tip)
37
{
38
   int i, j;
39
 
40
   // If we are placing a tooltip beyond our predefined matrix, 
41
   // create a bigger matrix and copy the old matrix over. This
42
   // lets us have a dynamic sized tooltip grid.
43
   if( (row >= rows) || (col >= cols) )
44
   {
45
       wxString* tmp = grid;
46
       grid = new wxString[(row+5) * (col+5)];
47
 
48
       for(i=0;i<rows;i++)
49
          for(j=0;j<cols;j++)
50
             grid[(i*cols)+j].Printf(wxT("%s"), tmp[(i*cols)+j]);
51
       delete[] tmp;
52
       rows = row+5;
53
       cols = col+5;
54
   }
55
   // We don't want to store a pointer to the user supplied tooltip,
56
   // we keep our own copy in the grid.
57
   grid[(row*cols)+col].Printf(wxT("%s"),tip);
58
}
59
 
60
int ttListCtrl::GetTooltip(int row, int col, wxString& tip)
61
{
62
   if( &grid[(row*cols)+col] != NULL )
63
      tip.Printf(wxT("%s"), grid[(row*cols)+col]);
64
}
65

This code does the following:

• Gives us a method to add tooltips to the X/Y (row/col) of the ttListCtrl
• Gives us a method to retrieve that tooltip
• Doesn’t require us to specify dimensions, it will automatically expand as our grid does

That last one was a big step for me. I add information to the grid as it comes in, so I have no idea how large my grid will get.
Next we have to listen for the mouse event, and determine the row/col the mouse is over. Here is the code, and I’ll talk about it.

 C++ | 
 copy code |
?
 
01
// cut from frame with ttListCtrl object
02
ttListCtrl* ListCtrlObject = new ttListCtrl(this, wxID_ANY, wxDefaultPosition, wxSize(350,250), wxLC_REPORT | wxLC_HRULES | wxLC_SINGLE_SEL);
03
 
04
ListCtrlObject->Connect(wxEVT_MOTION, wxMouseEventHandler(ttListCtrl::OnMouseMotion));
05
// cut
06
 
07
/* ttListCtrl.cpp */
08
void ttListCtrl::OnMouseMotion(wxMouseEvent& event)
09
{
10
   ttListCtrl* o = (ttListCtrl*)event.GetEventObject();
11
   if( o == NULL ) return;
12
   int id = event.GetEventType();
13
   int count = o->GetColumnCount();
14
   long row=-1, col=-1;
15
   int flags=0, i=0, totalWidth=0, tmpWidth=0;
16
   wxPoint pt;
17
   wxTipWindow* tipWin;
18
   wxTimer* killTip;
19
   wxString toolTip;
20
 
21
   if( id != wxEVT_MOTION )
22
	   return;
23
 
24
   // We can get the row easily(HitTest) but the columns are more tricky.
25
   // We get all the column widths, and once our total width is beyoind where the
26
   // mouse is, we know which column the mouse is over.  This lets us resize the
27
   // columns during runtime and still get correct tooltips.
28
   pt = event.GetPosition();
29
   row = o->HitTest(pt, flags);
30
   for(i=0;i<count;i++)
31
   {
32
       tmpWidth = o->GetColumnWidth(i);
33
       totalWidth += tmpWidth;
34
       if( pt.x < totalWidth )
35
       {
36
            col = i;
37
            break;
38
       }
39
   }
40
 
41
   // If things look valid, get the tooltip
42
   if( row > -1 && col > -1)
43
	   o->GetTooltip(row, col, toolTip);
44
 
45
   // If we have a tooltip, we want to show is for 1 second, and then disappear. 
46
   // We use a timer for this.
47
   if( toolTip.Length() > 0 )
48
   {
49
      tipWin = new wxTipWindow(o,toolTip);
50
      // Bind() is only avail for 2.9.0 and later
51
      //Bind(wxTimerEvent, Monitor::destroyTip, wxID_ANY, wxID_ANY, q);
52
      SetClientData(tipWin);
53
      Connect(wxEVT_TIMER, wxTimerEventHandler(ttListCtrl::destroyTip), NULL, this);
54
      killTip = new wxTimer(this, wxID_ANY);
55
      killTip->Start(1000, true);
56
   }
57
}
58
 
59
void ttListCtrl::destroyTip(wxTimerEvent& event)
60
{
61
   wxTipWindow *obj = (wxTipWindow*)GetClientData();
62
   if( obj ) obj->Destroy();
63
}
64

This event methods will solve the rest of our problems:

• Listen for mouse movement over a cell
• Determine the row/col the mouse is over
• Display the proper tooltip and automatically destroy the window

Now all we need to do is create our ttListCtrl object, add columns and rows, assign the tooltips, and connect a wxEVT_MOTION to the ttListCtrl. Then we have tooltips! Here is a quick example of how to use the class, roughly based on my XBee Monitor program(next posting)

 C++ | 
 copy code |
?
 
01
    packetList = new ttListCtrl(this, wxID_ANY, wxDefaultPosition, wxSize(350,250), 
02
         wxLC_REPORT | wxLC_HRULES | wxLC_SINGLE_SEL);
03
    wxListItem item;
04
 
05
    item.SetText(_("Type"));
06
    packetList->InsertColumn(0, item);
07
    item.SetText(_("Size"));
08
    packetList->InsertColumn(1, item);
09
    item.SetText(_("Network"));
10
    packetList->InsertColumn(2, item);
11
 
12
    packetList->SetColumnWidth(0, wxLIST_AUTOSIZE_USEHEADER);
13
    packetList->SetColumnWidth(1, wxLIST_AUTOSIZE_USEHEADER);
14
    packetList->SetColumnWidth(2, wxLIST_AUTOSIZE_USEHEADER);
15
 
16
    wxListItem _item;
17
    wxString tt;
18
 
19
    _item.SetId(0);
20
    _item.SetColumn(0);
21
    _item.SetText(wxT("acol 1"));
22
    tt.Printf("acol1 Tooltip");
23
    packetList->SetTooltip(0, 0, tt);
24
    packetList->InsertItem(_item);
25
 
26
    _item.SetId(0);
27
    _item.SetColumn(1);
28
    _item.SetText(wxT("acol 2"));
29
    tt.Printf("acol2 Tooltip");
30
    packetList->SetTooltip(0, 1, tt);
31
    packetList->SetItem(_item);
32
 
33
// cut ...
34
 
35
    _item.SetId(2);
36
    _item.SetColumn(3);
37
    _item.SetText(wxT("ccol 4"));
38
    tt.Printf("ccol4 Tooltip");
39
    packetList->SetTooltip(2, 3, tt);
40
    packetList->SetItem(_item);
41
 
42
    packetList->Connect(wxEVT_MOTION, wxMouseEventHandler(ttListCtrl::OnMouseMotion));
43

And a screen shot of its success:

I’ve also attached the entire class just for completeness. I’m new to wxWidgets, so there may be a better way tha tI would love to hear.

 C++ | 
 copy code |
?
 
001
/* ttListCtrl.h */
002
/* Ryan Day, http://www.ryanday.net/ */
003
 
004
#include <wx/timer.h>
005
#include <wx/tipwin.h>
006
#include <wx/listctrl.h>
007
 
008
class ttListCtrl : public wxListCtrl
009
{
010
private:
011
   wxString *grid;
012
   int cols, rows;
013
 
014
public:
015
   ttListCtrl() { };
016
   ttListCtrl( wxWindow *parent,
017
                wxWindowID winid = wxID_ANY,
018
                const wxPoint& pos = wxDefaultPosition,
019
                const wxSize& size = wxDefaultSize,
020
                long style = wxLC_REPORT,
021
                const wxValidator& validator = wxDefaultValidator,
022
                const wxString &name = wxListCtrlNameStr)
023
   {
024
       cols = rows = 3;
025
       grid = new wxString[3*3];
026
       Create(parent, winid, pos, size, style, validator, name);
027
   }
028
 
029
   int SetTooltip(int row, int col, wxString& tip);
030
   int GetTooltip(int row, int col, wxString& tip);
031
 
032
   void OnMouseMotion(wxMouseEvent& event);
033
   void destroyTip(wxTimerEvent& event);
034
};
035
 
036
/* ttListCtrl.cpp */
037
/* Ryan Day, http://www.ryanday.net/ */
038
 
039
#include "ttListCtrl.h"
040
 
041
int ttListCtrl::SetTooltip(int row, int col, wxString& tip)
042
{
043
   int i, j;
044
 
045
   if( (row >= rows) || (col >= cols) )
046
   {
047
	   wxString* tmp = grid;
048
	   grid = new wxString[(row+5) * (col+5)];
049
 
050
       for(i=0;i<rows;i++)
051
          for(j=0;j<cols;j++)
052
             grid[(i*cols)+j].Printf(wxT("%s"), tmp[(i*cols)+j]);
053
       delete[] tmp;
054
       rows = row+5;
055
       cols = col+5;
056
   }
057
   grid[(row*cols)+col].Printf(wxT("%s"),tip);
058
}
059
 
060
int ttListCtrl::GetTooltip(int row, int col, wxString& tip)
061
{
062
   if( &grid[(row*cols)+col] != NULL )
063
      tip.Printf(wxT("%s"), grid[(row*cols)+col]);
064
}
065
 
066
void ttListCtrl::OnMouseMotion(wxMouseEvent& event)
067
{
068
   ttListCtrl* o = (ttListCtrl*)event.GetEventObject();
069
   if( o == NULL ) return;
070
   int id = event.GetEventType();
071
   int count = o->GetColumnCount();
072
   long row=-1, col=-1;
073
   int flags=0, i=0, totalWidth=0, tmpWidth=0;
074
   wxPoint pt;
075
   wxTipWindow* tipWin;
076
   wxTimer* killTip;
077
   wxString toolTip;
078
 
079
   if( id != wxEVT_MOTION )
080
	   return;
081
 
082
   pt = event.GetPosition();
083
   row = o->HitTest(pt, flags);
084
   for(i=0;i<count;i++)
085
   {
086
       tmpWidth = o->GetColumnWidth(i);
087
       totalWidth += tmpWidth;
088
       if( pt.x < totalWidth )
089
       {
090
    	   col = i;
091
    	   break;
092
       }
093
   }
094
 
095
   if( row > -1 && col > -1)
096
	   o->GetTooltip(row, col, toolTip);
097
 
098
   if( toolTip.Length() > 0 )
099
   {
100
      tipWin = new wxTipWindow(o,toolTip);
101
      //Bind(wxTimerEvent, Monitor::destroyTip, wxID_ANY, wxID_ANY, q);
102
      SetClientData(tipWin);
103
      Connect(wxEVT_TIMER, wxTimerEventHandler(ttListCtrl::destroyTip), NULL, this);
104
      killTip = new wxTimer(this, wxID_ANY);
105
      killTip->Start(1000, true);
106
   }
107
}
108
 
109
void ttListCtrl::destroyTip(wxTimerEvent& event)
110
{
111
   wxTipWindow *obj = (wxTipWindow*)GetClientData();
112
   if( obj ) obj->Destroy();
113
}
114

Ok I don’t know about you but whenever I have the chance to create what anybody labels a “robot” I totally flip out. So Google has “robots” that you can include in your wave conversations and I thought to myself “Finally a chance to create a robot this will be badass!”. I got as far as back propagation neural networks before I decided that, not only do I not know what I wanted my robot to do, but whatever it did I certainly wouldn’t need some sort of half assed AI to do it.

So I gave up with the robot thing and focused my newly found bpnn info on my job. Soon I realized two things:

1. We don’t even track the information I needed to properly train my network
2. It’s suddenly Friday and I still have to finish throwing more functionality into our spaghetti logic mess of a program.

So I again gave up and went camping.  Camping always brings several levels of victories.  I was victorious because I slept outside in the freezing rain. I suppose you could also call that a defeat since I only chose to sleep outside because the cabin was filled with wasps.

That is neither here nor there.  On with googlebots!

Googlebots have to be programmed in Python because Google only distributes the API in Java and Python and I hate Python just less then I hate Java.  If you look at the two APIs, you will probably choose Java because it(on first appearance anyway) appears to far outshine the Python version.  Anyways, I want a simple bot to help automate developer conversations. Specifically, to replace talk about a bug with links to that actual bug.  So here:

1. Monitor a wave and try to tell if the people are talking about a bug
2. Lookup that bug on the bugzilla page
3. Modify the bug text to link to the desired bug

Not very aggressive, but then again I’ve never written a Googlebot and I don’t know Python.  So what more do you want.  [Edit: as I went through this experience, it turned out to be more an exercise in learning Python then learning to design a google robot. I feel that the API is pretty simple and building a basic robot is not very complex]

To start we must setup our initial robot and monitor a wave.  This requires nothing more then the cookie cutter code Google provides, and listening to the BlipSubmitted event.

 Python | 
 copy code |
?
 
01
from waveapi import events
02
from waveapi import model
03
from waveapi import robot
04
 
05
import httplib
06
import re
07
import logging
08
 
09
def OnBlipSubmitted(properties, context):
10
  """Invoked when a blip is entered(the 'done' button is clicked)
11
 
12
def OnRobotAdded(properties, context):
13
  """Invoked when the robot has been added."""
14
 
15
if __name__ == '__main__':
16
  myRobot = robot.Robot('ryansrobot',
17
      image_url='http://ryansrobot.appspot.com/icon.png',
18
      version='3',
19
      profile_url='http://ryansrobot.appspot.com/')
20
  myRobot.RegisterHandler(events.BLIP_SUBMITTED, OnBlipSubmitted)
21
  myRobot.RegisterHandler(events.WAVELET_SELF_ADDED, OnRobotAdded)
22
  myRobot.Run()

Thats just about all there is to the wave code. When your robot is added to a Wave, OnRobotAdded will be run. Anytime a blip is submitted, OnBlipSubmitted is run.  Lets examine the OnRobotAdded method more indepth.

 Python | 
 copy code |
?
 
1
def OnRobotAdded(properties, context):
2
  """Invoked when the robot has been added."""
3
  root_wavelet = context.GetRootWavelet()
4
  root_wavelet.CreateBlip().GetDocument().SetText("Robot checking in")
5
  logging.info('robotAdded')
6

This is pretty straightforward.  You create a new blip in the wavelet your robot has been added to.  The new blip is appended to the end of the wavelet.  You set the text of your new blip’s document to say ‘Robot checking in’.  You can do neat things also, like create a new wavelet but not invite anyone to that wavelet.  Only your robot can access the wavelet now and can use it to store/retrieve information. We don’t need to do that though, so we just deal with the current wavelet.

So now what we do is monitor new blips and look for things that appear to be bugs.  As I mentioned before, I’m not bothering to do any half-assed AI in order to learn the Wave API, so I don’t do much to determine what is bug talk. I feel that is beyond the scope of this document.

 Python | 
 copy code |
?
 
01
def OnBlipSubmitted(properties, context):
02
  logging.info('blipSubmitted')
03
  blip = context.GetBlipById(properties['blipId'])
04
  contents = blip.GetDocument().GetText()
05
  pattern = re.compile('bug (\d+)', re.M | re.I)
06
  m = pattern.search(contents)
07
  if m:
08
        msg = getDescription(m.group(1))
09
        newmsg = re.sub(m.group(0), "<a href=\""+msg+"\">"+m.group(0)+"</a>", contents)
10
        waveId = blip.GetWaveId()
11
        waveletId = blip.GetWaveletId()
12
        if msg != "Not Found":
13
                blip.GetDocument().SetText(" ")
14
                context.builder.DocumentAppendMarkup(waveId, waveletId, blip.GetId(), newmsg)
15
 
16
def getDescription(term):
17
  logging.info(term);
18
  host = "bugzilla.mozilla.org"
19
  h = httplib.HTTPS(host)
20
  h.putrequest('GET', '/buglist.cgi?quicksearch='+term)
21
  h.putheader('Host', host)
22
  h.putheader('User-agent', 'Ryans Robot')
23
  h.endheaders()
24
  returncode, returnmsg, headers = h.getreply()
25
  if returncode == 302:
26
        return headers.getheader('Location')
27
  else:
28
        return 'Not Found'
29

Non wave: Look through the blip’s document for text matching r’bug (\d+)’.  Search for that \d+ on your bugzilla site(in this case Mozilla) and return a link.

Wave: Now change the blip’s document to be the original text, but with with the anchor tag attached to the bug text.  This is a little more complicated with the Python API then with the Java API.  Java has a .appendMarkup() method, which we don’t have in Python.  What we do is use a direct builder method to create the JSON that gets sent back to the Wave server.  This lets us create bold tags and anchor tags, and as long as you have well formed HTML you can put whatever in the blip’s document.

First we erase whatever text was in the document, then we replace the document with the original text, including the anchor tag.  This gives us a nice link the the actual bug.

The above code that adds html tags to the blip is really the last wavy part of the robot. The rest is python parsing and other things, which I’m not good at and am in no position to explain. The code works on Mozilla’s bugzilla page, so I’ll paste that here. 

Wave can be used on your local network so my goal here is to allow my company to discuss bugs on our Wave server, and provide live links to those bugs on our bugzilla site.  Maybe somebody reading this will be able to help me out with the Python, which I’m sure is attrocious. 

 Python | 
 copy code |
?
 
01
# http://www.ryanday.net/
02
# search bugzilla by bug ID, replace blip text with link
03
 
04
from waveapi import events
05
from waveapi import model
06
from waveapi import robot
07
 
08
import httplib
09
import re
10
import logging
11
logging.info('base')
12
 
13
import BaseHTTPServer, urlparse, sys
14
 
15
def OnParticipantsChanged(properties, context):
16
  """Invoked when any participants have been added/removed."""
17
  logging.info('participantsChanged')
18
  added = properties['participantsAdded']
19
  for p in added:
20
    Notify(context)
21
 
22
def OnBlipSubmitted(properties, context):
23
  logging.info('blipSubmitted')
24
  blip = context.GetBlipById(properties['blipId'])
25
  contents = blip.GetDocument().GetText() 
26
  pattern = re.compile('bug (\d+)\n?', re.M | re.I)
27
  m = pattern.search(contents)
28
  if m:
29
 msg = getDescription(m.group(1))
30
 newmsg = re.sub(m.group(0), "<a href=\""+msg+"\">"+m.group(0)+"</a>", contents)
31
 waveId = blip.GetWaveId()
32
 waveletId = blip.GetWaveletId()
33
 if msg != "Not Found":
34
 blip.GetDocument().SetText(" ")
35
 context.builder.DocumentAppendMarkup(waveId, waveletId, blip.GetId(), newmsg)
36
  wavelet = context.GetRootWavelet()
37
 
38
def getDescription(term):
39
  host = "bugzilla.mozilla.org"
40
  h = httplib.HTTPS(host)
41
  h.putrequest('GET', '/show_bug.cgi?id='+term)
42
  h.putheader('Host', host)
43
  h.putheader('User-agent', 'Ryans Robot')
44
  h.endheaders()
45
  returncode, returnmsg, headers = h.getreply()
46
  if returncode == 200:
47
 body = h.getfile().read()
48
 regex = '<title>Bug ' + term + ' '
49
 pattern = re.compile(regex, re.M)
50
 m = pattern.search(body)
51
 if m:
52
 return 'https://bugzilla.mozilla.org/show_bug.cgi?id='+term
53
 else:
54
 return 'Not Found'
55
 
56
def OnRobotAdded(properties, context):
57
  """Invoked when the robot has been added."""
58
  root_wavelet = context.GetRootWavelet()
59
  root_wavelet.CreateBlip().GetDocument().SetText("Robot checking in")
60
  logging.info('robotAdded')
61
 
62
def Notify(context):
63
  root_wavelet = context.GetRootWavelet()
64
  root_wavelet.CreateBlip().GetDocument().SetText("&lt;dr. nick&gt;Hi everybody!&lt;/dr. nick&gt;")
65
 
66
if __name__ == '__main__':
67
  logging.info('main')
68
  myRobot = robot.Robot('ryansrobot', 
69
      image_url='http://ryansrobot.appspot.com/icon.png',
70
      version='3',
71
      profile_url='http://ryansrobot.appspot.com/')
72
  myRobot.RegisterHandler(events.BLIP_SUBMITTED, OnBlipSubmitted)
73
  myRobot.RegisterHandler(events.WAVELET_SELF_ADDED, OnRobotAdded)
74
  myRobot.Run()
75

Theres some search engine fodder!! So Eric Schmidt says

“If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place.”

I fucking hate that point of view. And here is why:

1. If I’m looking for a new job, I don’t want my current company to know. Should I not be looking for a new job?
2. If I’m developing Tor, or Tor related software, but my company is owned by a country that wants to ban privacy software, should I not be working on Tor?
3. If I like other guys, but am in the Army, should I not be in the Army, or not be gay? Which one are you against?

Any of these points can be countered by saying I shouldn’t publicize what I’m doing if its touchy.  But Google makes things publicly accessible in a way they might otherwise not be.

For instance, if I’m looking for a new job and don’t want my current employer to find out, I shouldn’t blast my resume all over Craigslist.  But what if I just post it to a few job sites?  Google aggregates that information in a way where my intentions may pop up by entering programming related words on my resume, and a location.

As for Tor development, I suppose I could use a handle instead of my real name. But doesn’t that still feed into the above quote?  But I have to use a handle, otherwise my name will be on the privacy software. If my name is on the privacy software, my employer could get upset.  What about Tor in the first place?  Yes, pedophiles use it. But gang bangers use guns.  If we ban Tor can we go ahead and ban guns too?

In 1776, guns brought this country it’s freedom.  Today Tor helps the people of fascist governments rise up.  The tools change, and if those tools offer the citizenry a way to unite then you can’t ban them.  I suppose I could keep going down this road and note that the Founding Fathers didn’t post flyers to invite the British to Philadelphia.  The Articles of Confederation weren’t posted for comment, with British response to be addressed to Washington.  That was done in secret.  I suppose that shouldn’t have happened either?

I really don’t think that I’m overstating the argument.  In Iran or China you have to do thing in secret, otherwise the government will get you.  So what about all my Republican friends who “weren’t doing anything wrong”, so they didn’t mind the loss of privacy from the Patriot Act?  Now they are pretty pissed about Obama’s computer privacy shitbill.  That brings things a little closer home.  But thats what you get! Never give up privacy, because later on someone you don’t like will use it against you!

I feel bipartisan in my privacy concerns.  I don’t give a shit who you are, stop making my right to privacy look like criminal intent.

You … stupids!

There is a new ObfuscaTOR wordpress plugin available!  I’ve added RSS feed support.  Don’t know if that will ever take off, but it seemed like a good way to provide bridge info from sources other then bridges.torproject. I also threw in some more config options for the widgets. You still have to configure the shortcode from the main Settings page though.

I also have had a chance to work on some embedded stuff at work.  I needed a way to display hex as ascii readable characters, and the libraries I’m using obviously don’t have sprintf(), so I came up with this:

 C | 
 copy code |
?
 
01
int convert(char *dst, char *src, int len) {
02
        char *ptr;
03
        int i;
04
 
05
        ptr = dst;
06
        for(i=0;i<len;i++) {
07
                *ptr = ((src[i] >> 4) & 0x0f);
08
                if( *ptr <= 9 ) *ptr += 48;
09
                else *ptr += 55;
10
                ptr++;
11
                *ptr = src[i] & 0x0f;
12
                if( *ptr <= 9 ) *ptr += 48;
13
                else *ptr += 55;
14
                ptr++;
15
        }
16
        *ptr = 0x0;
17
}

This function would convert a string, say, “\xde\xad\xbe\xef” to “DEADBEEF”.  Its been a loooooong time since I’ve gotten to use C, so I’m happy I still remember how(at least I think I do).  Obviously I’m not doing any error checking at all there, but as I remember thats pretty standard. You have to make sure all your buffers are big enough before you ever call something.

Wow so sometimes I know I’m learning things again when I can feel a sensation in my head almost like the feeling my little nothing bicep gets from lifting my coffee mug. Postgres’ internal database structure is fucked up and I’ll let you discover the relationship between the two data sets.

The goal:

I want to make a pretty basic postgres query builder for my job. We always have to make simple, custom reports for clients.  They are basic, but take up our time.  This query builder has to be very user friendly and allow them to create their own reports.

It’s a pretty simple task(none of these reports are expected to have any logic beyond basic where clauses, ie – no joins), but having to look through the postgres documentation and remember that pg_class.oid = pg_constraint.conkey unless its the foreign relation whence you use confkey unless a bunch of other stuff also from pg.att and pg_constraint blah just isn’t what I was designed to keep in my head.

My main problem is this:

I want users to be able to say “Show me all Terminal Names and Location Names for Location #7″, but I can’t expect the users to know about a Terminal table and a Location table, and much less which fields relate those two tables.

Well, setting the location=7 is easy, but I can’t ask users to manually specify Terminal.Location = Location.id && Location.id = 7. The user just knows to say “Location 7″. So I relate the tables through foreign keys. I look at all the tables involved (terminals and locations in this case) and find out how they are related. After some reading and experimenting,  this query relates tables to one another by looking through the foreign keys.

 MySQL | 
 copy code |
?
 
1
SELECT pc1.relname as ltable, pga2.attname as lcolumn, pc2.relname as rtable, pga1.attname as rcolumn
2
                FROM pg_class pc1, pg_class pc2, pg_constraint, pg_attribute pga1, pg_attribute pga2
3
                WHERE pc1.relname in ({$sourceList}) and pg_constraint.conrelid = pc1.oid
4
                AND pc2.oid = pg_constraint.confrelid
5
                AND pc2.relname in ({$destList})
6
                AND pga1.attnum = pg_constraint.confkey[1]
7
                AND pga1.attrelid = pc2.oid
8
                AND pga2.attnum = pg_constraint.conkey[1]
9
                AND pga2.attrelid = pc1.oid

oh noes my code plugin has MySQL code but no Postgres code… no time keep going.

The ’sourceList’ represents the table(s) you want to relate to ‘destList’. If the two lists are equal(containing the same tables) then this query just shows all the links to each other:

 MySQL | 
 copy code |
?
 
1
    ltable     |    lcolumn   |        rtable         |   rcolumn
2
--------------+---------------+-------------------+-------------
3
Terminal | location_id | Locations | location_id

Now I can manually add a WHERE clause to relate Terminal.location_id to Locations.location_id.

But now it gets fun! We have routes!

Routes HasMany Locations. Locations HasMany Terminals. Terminals have no direct relation to Routes.

So when a user says “Show me Terminal Names for Route #2″, I don’t have a direct relation between the two tables! The solution is obvious, get all foreign keys for the Terminals table. One will relate to the Locations table. Get all foreign keys for the Locations table, one will relate to Route. Then I can create the link manually.

This probably should be done recursively, but I don’t do it that way for a couple reasons.

1. Go do it yourself, I’m hungry and going to lunch.
2. Second though is I have no idea how all the relations in our database work out. I could end up in a loop and not really get out of it, it would take a lot of time, etc.

So I just go one level. If I can’t link your two tables through a single third table, then you’re SOL. I run this query:

 MySQL | 
 copy code |
?
 
1
SELECT pc1.relname as ltable, pga2.attname as lcolumn, pc2.relname as rtable, pga1.attname as rcolumn
2
                        FROM pg_class pc1, pg_class pc2, pg_constraint, pg_attribute pga1, pg_attribute pga2
3
                        WHERE pc1.relname = '{$table}' and pg_constraint.conrelid = pc1.oid
4
                        AND pc2.relkind = 'r' AND pc2.oid = pg_constraint.confrelid
5
                        AND pga1.attnum = pg_constraint.confkey[1]
6
                        AND pga1.attrelid = pc2.oid
7
                        AND pga2.attnum = pg_constraint.conkey[1]
8
                        AND pga2.attrelid = pc1.oid

This gives me all the foreign keys for a table, say Terminal. Then I run the first query on all the foreign tables(rtable from the first query) to see if I can link to the desired table (Route). If I can awsome! If not, move on.

That concludes my adventure is dealing with the internal Postgres information schema.  I hope this will save me a couple hours of trouble further down the road.  I did move on to pl/pgsql, and that is a huge pain in the neck too… but maybe for another day.

Please try out the Wordpress obfuscaTOR plugin!  I’ve tested it on Linux and Windows, all you need are the GD libraries. Fonts are included in the library, still waiting for a way to make that work well without including the fonts(need ideas!).  

If you choose to run this plugin you are helping to distribute Tor bridge information.  This can be used by many people to access information restricted by their government, or to remain anonymous when posting political information to the internet.  It’s hard to distribute this information because listing the IP addresses on a site will either get that one site blocked, or have all those IP addresses filtered.  But if the information is obfuscated, computers can’t read it(most of the time) so filtering can’t be automated.  If tons of blogs display this obfuscated information, then you can block any one particular distribution point.  This will hopefully give more people the ability to remain anonymous while preventing governments from censoring the information.

Also check out the Tor Project and consider running one of these relays.  If people don’t run relays, then there is no bridge information to distribute.

The idea is to distribute bridge information to as many people as possible, without a censoring party being able to automate the collection and filtering of those bridges. So what better way then to pull bridge information, use some captcha libraries, and write plugins for popular blogs/CMS/etc so everyone can use it?  I’ve got a Wordpress pluging(bottom right on the categories list, dunno where else to put it really…) and my friend Alex Saify is writing a Drupal plugin as we speak.

If you can’t tell what the heck the bridge is, please reload the page.  One of the captcha algorithms isn’t working quite right yet.

We put together a PHP library that uses some popular captcha libs, and provides and easy interface to extend and implement those libraries. Right now we are debugging everything, but will release ASAP.  I also have to make sure caching is working correctly in the plugins, otherwise there is a risk of flooding the bridges server by accident.

Where to get bridge information?

1. bridges.torproject.org
2. RSS feeds
3. Twitter feeds

So far I only use #1.  But the rest are on the way as well.  I noticed I made it onto the tor blog, so I wanted to get this post up quick.  I’m making it very easy to integrate your custom Captcha algorithms into this library. But the more algorithms that can be included by default the better.  Also, if you notice that the algorithms I have are easy to defeat, please tell me!!  I don’t want to include a captcha algorithm that is too easy to crack, otherwise the purpose is defeated.