function url_decode(str){
var hex = /^[0-9a-fA-F]{2}/;
var out='';
var arr = str.split('%');
if(arr.length<2) return str;
for(var i=0;i<arr.length;i++)
{
  /*look for hex values */
  if(hex.exec(arr[i])) {
    out += String.fromCharCode(parseInt(arr[i].substring(0,2),16))+arr[i].substring(2,arr[i].length);
  } else { if(i==0) out+=arr[i]; else out+='%'+arr[i];
  }
}
return utf8to16(out);
}

function utf16to8(str) {
    var out, i, len, c;

    out = "";
    len = str.length;
    for(i = 0; i < len; i++) {
	c = str.charCodeAt(i);
	if ((c >= 0x0001) && (c <= 0x007F)) {
	    out += str.charAt(i);
	} else if (c > 0x07FF) {
	    out += String.fromCharCode(0xE0 | ((c >> 12) & 0x0F));
	    out += String.fromCharCode(0x80 | ((c >>  6) & 0x3F));
	    out += String.fromCharCode(0x80 | ((c >>  0) & 0x3F));
	} else {
	    out += String.fromCharCode(0xC0 | ((c >>  6) & 0x1F));
	    out += String.fromCharCode(0x80 | ((c >>  0) & 0x3F));
	}
    }
    return out;
}

function utf8to16(str) {
    var out, i, len, c;
    var char2, char3;

    out = "";
    len = str.length;
    i = 0;
    while(i < len) {
	c = str.charCodeAt(i++);
	switch(c >> 4)
	{
	  case 0: case 1: case 2: case 3: case 4: case 5: case 6: case 7:
	    // 0xxxxxxx
	    out += str.charAt(i-1);
	    break;
	  case 12: case 13:
	    // 110x xxxx   10xx xxxx
	    char2 = str.charCodeAt(i++);
	    out += String.fromCharCode(((c & 0x1F) << 6) | (char2 & 0x3F));
	    break;
	  case 14:
	    // 1110 xxxx  10xx xxxx  10xx xxxx
	    char2 = str.charCodeAt(i++);
	    char3 = str.charCodeAt(i++);
	    out += String.fromCharCode(((c & 0x0F) << 12) |
					   ((char2 & 0x3F) << 6) |
					   ((char3 & 0x3F) << 0));
	    break;
	}
    }
    return out;
}

function trace(url){
var tracker = _gat._getTracker("UA-XXXXXXX-X");
tracker._trackPageview(url);
}

Now every time I need to track something I call this function with a custom build URL.

CREATE TABLE pages
(
  id bigint NOT NULL,
  url character varying(255),
  md5 character(32),
  CONSTRAINT pages_pkey PRIMARY KEY (id)
)

The faster lookup should mainly be possible through the shorter column length (and therefore smaller index). Actually I don’t know if the fixed width is good or bad here, but hashes usually don’t vary in length. After creating this table I added a B-Tree unique Index to the md5 column to enable a fast lookup.
After a while a noticed a rather high CPU load on lookups for this table so I tried to analyze the problem. First I tried the obvious through psql.

cloud=# explain analyze select * from pages where md5 ='abc';
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Index Scan using i_pages_md5 on pages  (cost=0.00..8.50 rows=1 width=166) (actual time=0.046..0.046 rows=0 loops=1)
   Index Cond: (md5 = 'abc'::bpchar)
 Total runtime: 0.157 ms
(3 rows)

As the explain shows all works perfectly fine and lookups shouldn’t be a problem. So there had to be something different what was going on.

After that I tried the same select with an actual md5.

cloud=# explain analyze select * from pages where md5 = md5('abc');
                                                  QUERY PLAN
--------------------------------------------------------------------------------------------------------------
 Seq Scan on pages  (cost=0.00..32017.63 rows=3994 width=166) (actual time=1203.699..1203.699 rows=0 loops=1)
   Filter: ((md5)::text = '900150983cd24fb0d6963f7d28e17f72'::text)
 Total runtime: 1203.769 ms
(3 rows)

Now you can see the plan does change quite a lot. I have a full table scan instead of an index scan. You can also see that the query time increases nearly by factor ten thousand.

The reason for this dramatic change is a simple type mismatch. For whatever reason the md5 function will be evaluated to a string of the type text. To create a match with the column md5 all values had to be casted to that type. The side effect of this is that the index can no longer be used, because it is of the wrong type.

To solve this I just had to cast the result of the md5 function back to something that is compatible with the index type. In my case I used a fixed width character field which is represented in the database as bpchar (blank padded character). So after modifying the query to the following I was back on index usage.

cloud=# explain analyze select * from pages where md5 = md5('abc')::bpchar;
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Index Scan using i_pages_md5 on pages  (cost=0.00..8.50 rows=1 width=166) (actual time=0.141..0.141 rows=0 loops=1)
   Index Cond: (md5 = '900150983cd24fb0d6963f7d28e17f72'::bpchar)
 Total runtime: 0.199 ms
(3 rows)

A lot of people out there use services like delicious where you tag you favorite sites and make this available to other users. Now I started to grab that data and began to build a massive tag cloud. After some time the site collected hundreds of thousands of links with their corresponding tags. So now you can start on the site and search for tags which interests you. These search tags will then be correlated against the cloud database and you get the most active links for your tags. So here is an example.
Let’s say you are interested in a tomcat tutorial.
tags:

tutorial, tomcat

results:

howtoforge.com
howtogeek.com
java.sun.com

Of course those results will be a link to the concrete tutorial (not just the entry page).
So far to the official part. For me this is more of a fun project. I prefer to start with some random tag and then wander around. It’s more like browsing, cause you start at points that you don’t already know. You have the chance to break out of your existing network of most used sites and see something new.

So have fun with it.

www.cloudtheweb.com

PS: For the implementation part – if you have any questions, just ask. I’m planning to explain some details about how it works on some later posts.

So enjoy the new features and lets hope the next release will not take another year.

First download the GLUEscript source from sourceforge.
A second tool you will need to get this running is premake. This is also a sourceforge project (you can use the binary version of it right away).
After downloading, I copied the premake binary into the glue/src folder.

So now we can start with fetching the dependencies which debian can fulfill.

sudo apt-get install libnspr4-0d libnspr4-0d-dbg libnspr4-dev libcurl4-openssl-dev libwxgtk2.8-dev libssl-dev libiodbc2-dev libmysql++-dev

In addition to that I needed a library called poco version 1.3.5 (all repositories I found just provided versions up to 1.3.3 -> those don’t work). So get the source from http://pocoproject.org/download/ (the complete version). Compiling that should make no trouble cause all the dependencies are already installed.

/tmp$ cd poco-1.3.5-all/
/tmp/poco-1.3.5-all$ ./configure
Configured for Linux
/tmp/poco-1.3.5-all$ make
/tmp/poco-1.3.5-all$ sudo make install

Now let’s get back to configuring GLUEscript. All configuration is done via lua script which will than be consumed by premake. The config file I needed to edit was the premake.lua file:

-- Check NSPR
if ( string.len(nspr_dir) == 0 ) then
  print("Using the NSPR library which is part of GLUEscript")
  dopackage("nspr") -- build our own NSPR
  nspr_dir = "../nspr/include"
  nspr_lib = "nspr"
  nspr_lib_dir = project.libdir
else
  print("You are using your own NSPR library: ")
  nspr_dir = "/usr/include/nspr"
  print("nspr include: " .. nspr_dir)
  print("nspr lib: " .. nspr_lib_dir .. "/" .. nspr_lib)
end

I copied the whole paragraph to just make it easier to find the position. Important is the added row in the else part.

nspr_dir = “/usr/include/nspr”

This is needed because debian has a different file structure for header files than the script expects it.

After that we are done with configuring. To actually start the build process you have to run premake first.

./premake gnu
make

The output will be generated to the following directory:

glue/bin/Debug

So far the makefile does not a an installation part. So if you want to install this you have to do it by yourself.

PS: This only works for the 0.0.1 version. So far I didn’t get any more recent svn version running.

CREATE TABLE tags
(
  id bigint NOT NULL,
  "value" character varying(150),
  CONSTRAINT tags_pkey PRIMARY KEY (id),
  CONSTRAINT tags_value_key UNIQUE (value)
)

CREATE TABLE sites_tags
(
  sites_id bigint NOT NULL,
  pages_id bigint NOT NULL,
  tags_id bigint NOT NULL,
  count integer,
  updated timestamp without time zone,
  CONSTRAINT sites_tags_pkey PRIMARY KEY (sites_id, pages_id, tags_id)
)

As you can see, the tags table is a simple value-id-table. The second table represents a join table between pages and tags.

The goal of my Query should be to get the most used tag from the join table. Only the first x-Rows are of interest to me. To get there I used a simple limit command. So just for comparison here a simple query of the join table without the actual values.

select sum(st.count) as anzahl from sites_tags st group by st.tags_id order by anzahl desc limit 50;
                                                                QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=13185.22..13185.35 rows=50 width=12) (actual time=1974.893..1975.033 rows=50 loops=1)
   ->  Sort  (cost=13185.22..13192.27 rows=2819 width=12) (actual time=1974.888..1974.941 rows=50 loops=1)
         Sort Key: (sum(count))
         Sort Method:  top-N heapsort  Memory: 18kB
         ->  HashAggregate  (cost=13056.34..13091.58 rows=2819 width=12) (actual time=1766.681..1876.092 rows=66136 loops=1)
               ->  Seq Scan on sites_tags st  (cost=0.00..10202.56 rows=570756 width=12) (actual time=0.120..690.719 rows=570756 loops=1)
 Total runtime: 1975.669 ms
(7 rows)

This is just a statement to get you the picture of cost for a simple query (without fetching any actual values).

To make this query useful I needed to add the values. All the values will be joined through the tags table.

Here the first implementation I came up with.

select t.value,sum(st.count) as anzahl from sites_tags st inner join tags t on t.id=st.tags_id group by st.tags_id,t.value order by anzahl desc limit 50;
                                                                      QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=123002.69..123002.82 rows=50 width=22) (actual time=12640.100..12640.239 rows=50 loops=1)
   ->  Sort  (cost=123002.69..124429.58 rows=570756 width=22) (actual time=12640.095..12640.153 rows=50 loops=1)
         Sort Key: (sum(st.count))
         Sort Method:  top-N heapsort  Memory: 20kB
         ->  GroupAggregate  (cost=91200.58..104042.59 rows=570756 width=22) (actual time=10165.002..12537.121 rows=66136 loops=1)
               ->  Sort  (cost=91200.58..92627.47 rows=570756 width=22) (actual time=10162.562..11564.604 rows=570756 loops=1)
                     Sort Key: st.tags_id, t.value
                     Sort Method:  external merge  Disk: 18808kB
                     ->  Hash Join  (cost=1877.06..24921.63 rows=570756 width=22) (actual time=259.674..3080.093 rows=570756 loops=1)
                           Hash Cond: (st.tags_id = t.id)
                           ->  Seq Scan on sites_tags st  (cost=0.00..10202.56 rows=570756 width=12) (actual time=0.070..781.449 rows=570756 loops=1)
                           ->  Hash  (cost=1050.36..1050.36 rows=66136 width=18) (actual time=259.518..259.518 rows=66136 loops=1)
                                 ->  Seq Scan on tags t  (cost=0.00..1050.36 rows=66136 width=18) (actual time=0.027..115.197 rows=66136 loops=1)
 Total runtime: 12647.403 ms
(14 rows)

As you can see, simply joining the table makes this query quite complex. The part which consumes most of the cost is the more complicated group by clause. Now the execution engine has to join these tables and then sort all values by id and string (mostly the value is the important part).

To avoid this there only could be one solution – remove the join. With removing the join there comes the question how to get the values from the second table. One way to do this would be to use the program (in my case a php web application) to query again for every line of the result set.
Another way to approach this would be to do a sub-select in the select section. This way you don’t have the additional round trip of doing it in the application. Another advantage would be that the database would only do these sub-selects for the actually returning rows (with respect of the limit).

So here the query I came up with (with the query execution plan)

select (select value from tags t where t.id=st.tags_id),sum(st.count) as anzahl from sites_tags st group by st.tags_id order by anzahl desc limit 50;
                                                                QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=36511.94..36512.07 rows=50 width=12) (actual time=2682.650..2682.790 rows=50 loops=1)
   ->  Sort  (cost=36511.94..36518.99 rows=2819 width=12) (actual time=2682.645..2682.705 rows=50 loops=1)
         Sort Key: (sum(st.count))
         Sort Method:  top-N heapsort  Memory: 20kB
         ->  HashAggregate  (cost=13056.34..36418.30 rows=2819 width=12) (actual time=1752.934..2570.690 rows=66136 loops=1)
               ->  Seq Scan on sites_tags st  (cost=0.00..10202.56 rows=570756 width=12) (actual time=0.109..713.541 rows=570756 loops=1)
               SubPlan
                 ->  Index Scan using tags_pkey on tags t  (cost=0.00..8.27 rows=1 width=10) (actual time=0.006..0.007 rows=1 loops=66136)
                       Index Cond: (id = $0)
 Total runtime: 2683.478 ms
(10 rows)

As you can see i still costs a lot. It is still 3 times more expensive then doing it without the values. On the other hand the cost is only a fourth of the cost of the join. This is mostly owed to the limit clause. The join has no way of knowing that it would be enough to run the limit without the join and later join the values. So far I found no way to tell postgres to do this more efficient.
So the simplest solution for that would be to do sub-queries. With that, the limit clause will be honored.

So as this example shows, it is always a good idea to try different approaches to one and the same query. Often you can see lots of differences in the execution plan which can have a major impact on performance.

CREATE OR REPLACE FUNCTION getHostFromUrl(p_url character varying)
  RETURNS character varying AS
$BODY$
declare
begin
  return substring(p_url from  'http.?://(.*?)/(.*)');
end;
$BODY$
  LANGUAGE 'plpgsql' VOLATILE
  COST 100;

So here a little tutorial how to make this work in the shell.

First choose your site.
I choosed this one (from a german stocks magazine I subscribed) just out of curiousity and I also have to mention, this site already has e-mail notification (so there is no need to actually use this on that site).

Second, get the XPath you want to extract. If you have knowledge of XPath this should be easy, if not use something like firebug to get there.

After that you can start on creating your XSL script for the transformation. The complete xslt should look something like this:

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="text"/>

<xsl:template match="/">
	<xsl:apply-templates select="*"/>
</xsl:template>

<xsl:template match="*">
	<xsl:apply-templates select="*"/>
</xsl:template>

<xsl:template match='//tr[@class="odd" and string-length(td[@class="date"])>0]'>
	<xsl:text>date: </xsl:text>
	<xsl:value-of select='td[@class="date"]'/>
	<xsl:text>,action: </xsl:text>
	<xsl:value-of select='td[@class="action"]'/>
	<xsl:text>,wkn: </xsl:text>
	<xsl:value-of select='td[@class="wkn"]'/>
	<xsl:text>,name: </xsl:text>
	<xsl:value-of select='td[@class="name"]/a'/>
	<xsl:text>,amount: </xsl:text>
	<xsl:value-of select='td[@class="amount"]'/>
	<xsl:text>,value: </xsl:text>
	<xsl:value-of select='td[@class="value"]'/>
	<xsl:text>
</xsl:text>
	<xsl:apply-templates/>
</xsl:template>

</xsl:stylesheet>

If you need more detail one XSLT you should check out w3schools. They have some good tutorials for starters.
The important part of this XSLT is the last template section. This is actually the part you have to use to get to your information. First comes the template match. Here you have to insert the XPath you have obtained before. After that you have to select (via XPath) what information you want and formulate how you want this to be written out.

Now you just have to put one and one together and you have your data mining solution.
I inserted the following bash script into my cron table and now have a subscription to this site.

curl -s http://www.deraktionaer.de/xist4c/web/Online---Musterdepot_id_1261_.htm | sed -e 's/&/&/g' - | xsltproc -html online.xslt -

Now that you have the raw information it should be no problem to get this into some mailinglist oder database for future use.

PS: In case you wonder about the sed in the statement. As I already mentionen the libXML is not really the most flexible solution for parsing HTML. Especially when it comes to HTML codes like ©, libXML resigns with an error. To avoid this I transformed all ampersands to escaped ampersands.

GTK
So on my end the complexity and structure of the existing framework is a necessity. What can and should be done is to clean up, remove deprecated functionality and maybe change little things to make the framework smoother.

File management
Today file management is mainly done through applications like nautilus and gnome-terminal. Managing information through folders and files is the default way since way back. For me this seems a perfectly fine way to handle information. On the other hand, with increasing storage capabilities I see more and more people wandering around the filesystem without any plan where to search for what information. There are multiple approaches to address this issue.
One would be to do everything through a search engine (like google desktop search, beagle). This would require that every peace of information would be crawlable and the indexer understands the structure and content. In reality this is not easily achievable. Most crawler just don’t understand the file format or can’t analyze the content correctly. If I have an text file in my home folder and it is a header file the information that I want from this file are completely different then if that file would be my mailbox backup in form of a mbox file. Todays crawler (at least the ones I have seen) ignore this fact and try to match strings against the plain content. This can help but is a very primitive way and the results are pretty random. So for me this seems not likely the way to go as a general approach.
Another approach could be to embrace a meta information level to give the user a custom structure on top of the filesystem. Basically this means just tag everything the way you want. This way you can create a different abstraction level where to find information. Mainly for MP3 this is the default way to organize music (ID3-tags). For pictures this is in creation through EXIF and XMP. Both are useful and standardized formats, but as always, let the format wars begin. For other file types it is pretty thin out there. So far I have seen no general system how to handle tags for binary files. It would be nice to see some progress on that end.
So there are quite a few things that could be done on the file management side but like in the gtk section the interesting question is what can be done in such short timeframe. For me, a tag system which plugs into nautilus (or other file managers) would be the ideal solution, because that way, you have the chance of bringing this to other desktop environments like KDE or (lets think big) maybe even Windows Explorer (Yes, it is extendable by API).

UI
My final point would be the UI and with UI I mean the complete environment starting with the desktop leading to application design. First let’s start with the desktop. It hasn’t changed since a very long time. So now there are two fractions out there.
One, everything should stay the same for ever.
Two, everything should be done some new way (although nobody states how).
For me both seam a bit drastic. The proposal to change the desktop to GnomeShell seams like a good compromise to me. You get a new interface but basically keep the classic applet bar and application menu structure. One side effect of GnomeShell is that it is implemented in JavaScript. So it should get easier to build some extensions for it (at least in theory). Another plus would be the integration of graphical effects in the system. I know that this is partly controversial because there are a lot of legacy system out there which can’t handle it, but for me eye-candy is always a good thing. So there should be at least some switch to disable the eye-candy for those who don’t want it (so far I didn’t see something like this during my first try).
A few weeks ago I was at a Microsoft conference (yes I know, evil) and they showed some of the new features of Windows 7. There was one thing which really impressed me. Aside from the new ‘superbar’ (former task bar) they could integrate widgets on the fly everywhere on the desktop. These widgets were partly some of the old stuff, known from vista’s sidebar, but some were quite interesting and could easily build via simple scripting. I would hope that Gnome would also bring some widget support which is not only usable in the gnome-applet bar. Especially the support for scripting languages as widget platform would really help to bring more developers to extending the desktop.
I know there are a lot of projects out there which have the goal of bringing gtk to different languages like Python, JavaScript, Perl, C#, only to name a few, but so far these are still relatively complex to use for simple widgets or applications.
So far I am no big fan of introducing another abstraction layer for designing applications (like XUL or XAML), but these systems encourage not so experienced developers to create new content (seen in the massive market of firefox plugins). So maybe it should be considered.

That’s it so far. For me it seams there is a lot to cover for the gnome 3.0 release. I think most of it is just visionary and not implementable in the short term. In the long term however I really would like to see some changes to the way users interact with computers (no, I do not mean multitouch).