Aranfoolcaytar

This day marks the third anniversary of my blog. Or at least, of the blog being moved from BlogSpot to my own domain, switching from being powered by Blogger to the Drupal CMS (in fact, the second Drupal site I ever installed and the oldest one that still remains), and of course being renamed from "Arancaytar's Little Corner" (a name that was exactly as cheesy as the posts) to "Aranfoolcaytar" (in other words, the blog is still cheesy but at least now it's a bit self-mocking).

Since that time, the blog has survived many trials: One change of hosts (to DreamHost), the best decision I ever made (and no, I didn't get paid for this), because friendly and tech-savvy support is worth way more than some three extra nines of uptime. Two separate major version upgrades in Drupal, from 4.7.x to 5.x to 6.x (and any Drupal user can attest to how much fun those are). My own increasingly invasive meddling with the core code as I learnt to develop for Drupal myself - though for the current code base, I managed to get back down to six core hacks. I tried out at least four different approaches to spam: Manual moderation, admin-approved registration required, Akismet auto-moderation, and finally ReCAPTCHA which rocks. Around 5-6 custom modules in various states of disrepair (though I can say I'm proud of how xbbcode turned out).

More theme changes than I can remember. My archives say I started out using the Slash theme that imitates Slashdot. I probably used Pleroma which I also used on the Grand Endeavor while it was still online. For a long time I simply used a green-shifted color mod (yay for color.module) of Garland, similar to what you can now see at proxydb). The last theme I invested much time in was Agregado which I eventually got rid of because of its poor support for fluid width and wide screens.

I didn't quite manage to break the 200 nodes mark at the same time, but at least I'm pretty close, this post being nid #196. Thanks to comment spam and the flood of notices generated by shoddily written contrib modules, the other two indicator numbers are the cid (comment ID) #1549, and the wid (error log ID) #1191949.

One of the more significant changes in this time has been that I finally found a desktop client I liked, namely Drivel. It was a bit tricky to get it to work with inline tags (you see, the Movable Type API separates post saving and taxonomy saving into two actions, and so Inline Tags has some trouble saving the inline taxonomy without getting it overwritten a second later), but it works very well.

Here's to three more years! I hope all my regular readers will stay with me that long - all three of you.

This is a bit of speculative brainstorming. A lot of this must have been thought of before, and when I have time I will research to see what papers have been written about it.

One premise is that Twitter's social network is currently being used in a different way than just a community of contacts. It is also a network through which messages spread virally, repeated and relayed from person to person. Casual users of Twitter may not have seen much of the informal so-called "RT" code yet. It is an abbreviation for "Re-Tweet", and is used interchangeably to signify that a certain message should be resubmitted by anyone who reads it, or that it already is a resubmission originally by someone else. This technique is used particularly in the #iran/#neda "tweetsphere", where news updates are globally significant rather than being personal communication between individuals.

A sample of what such updates may look like is here, picked up some minutes ago.

This acts as a mass-moderated, theoretically decentralized network spreading short messages between peers without requiring every broadcast to be sent to every person, allowing indirect messages to spread further the more significantly they are considered by the people who read them.

I say "theoretically" because in practice, all this still happens over the Twitter database. Twitter's social network is a virtual construct within the central entity of Twitter. "Retweeting" does not actually do anything like "passing on a message"; it merely produces a copy of the same message on the same server, readable by other subscribers. To block this communication, all a censor has to do is filter access to twitter.com (which is in fact happening in Iran, last I heard).

Yesterday around three in the morning, I spent some time feverishly wondering whether Twitter's virtual "relay network" on a single database could be turned into an actual decentralized relay network between a multitude of computers, highly resistant to filtering.

The threats that this network must be safe from are these:

Port blocking

The software must either communicate via randomized ports or via a port that cannot be blocked unilaterally (80).

Active Infiltration

The network must be resistant to malicious flooding/spamming. This would work via a web of trust where peers are gradually gaining more trust the more of their messages are passed on.

Attacking the center

Whatever mechanism the network uses to introduce peers to each other, no central database is safe from attack.

Passive Infiltration

The network must not allow peers to harvest peer identities, because privacy is a matter of life or death in Iran right now.

Protections against the first two threats are solidly established, and implemented in many Peer-to-Peer technologies in the wild, such as BitTorrent (which already has reputation networks for prioritizing those peers known to be the best contributors).

The second threat is somewhat tricky, as Blue Frog and more recently The Pirate Bay have shown. Blue Security, for those who missed it, used a peer-based approach to spam fighting three years ago. In May/June 2006, Blue Security was hit by vast DDoS attacks and eventually shut down: The central server, the vulnerable Achilles heel in the system, had been disabled. The Pirate Bay is presently embroiled in a lawsuit for "enabling" copyright infringement, and in spite of the existence of "trackerless Torrents" via a Distributed Hash Table, it is clear that without a central tracker like TPB, discovering an initial peer i less simple.

The last is even more intricate, as it makes discovery of peers barely possible. When you join a BitTorrent cloud, you subscribe to a tracker database that contains your IP and those of the other peers. An interested entity (such as the RIAA and its slightly more evil cousin, the Iranian secret police) can easily subscribe to the same tracker, discover those peers that happen to be within its sphere of influence and then act accordingly (sue for millions, beat to death with axes, et cetera). Those who use BitTorrent for copyright violations don't bother with more privacy, as legal proceedings in the US impose (some) rules on what constitutes evidence or ethical investigation. Basij on motorcycles with axes knocking on your door at night don't have such inhibitions, so people need hard crypto.

The basic dilemma is: how can a decentralized network grow dynamically, while no single computer is allowed access to all peers, and no central database is safe from attack? It seems like a chicken-and-egg problem. Some steganography ideas come to mind (hiding peer addresses in remote corners of the web or in spam mail), or some indirect routing ideas (eg. all connections inside a censoring country must first go outside that country across jurisdictional boundaries), but that's very theoretical. I don't know enough about network technology.

Brainstorm out.

In re my previous post, needless to say it is nearly impossible to properly operate a terminal if the background scripts interrupts each minute by injecting messages directly into the character console. Particularly when you are trying to edit a text file in vim at the moment.

I saw the light of reason and am now simply appending the messages to a text file. Then, I can open another terminal, set it to follow the text file using tail -f file. Whew. My shell is calm and peaceful again.

This is my first post with the new, custom-created article content type. There is very little to outwardly distinguish it from the blog content type provided by Drupal's blog.module, except that the latter does not have a link to "Arancaytar's blog" below each entry (which is superfluous seeing as I am the only author on this site, and my blog is equivalent to the front page).

I am exploring a way out of blog.module because I do follow Drupal's core development very closely, and I have the feeling that this module is becoming an unloved stepchild (and not without reason, since most of its features are hard-coded versions of what could be done with contributed modules like Views). Ultimately, since this is a single-user blog, the recommended content type for me is the article type (what used to be story in Drupal 5).

This post serves as a kind of prototype. If I can make sure article nodes have the same settings as blog nodes, I will begin experimenting with switching my old nodes to the new type all at once.

I've built myself a twitter notifications daemon out of duct-tape and spit (with liberal application of bash)!

First of all, here is the daemon script itself. It is in bash, and runs in a continuous loop until killed. I use several components, most notably "twidge" to download new messages. Twidge is a command line utility for twitter. When twidge receives new messages, I display them with notify-send.

#!/bin/bash
echo "Starting demon..."
 
while true
do
  echo "Downloading messages on "`date`
  messages=`twidge -c ~/.twidgecron lsrecent -alsu|tac|awk -F "t" '{print $2,$4}'`
  if [ ! -z "$messages" ]
  then
    echo "New messages!"
    for terminal in `ls /dev/pts`
    do
      echo "" /dev/pts/$terminal
      echo "New Tweets!" /dev/pts/$terminal
    done
 
    echo "$messages"|while read message
    do
      message=($message)
      sender=${message[0]}
      icon=/usr/share/icons/gnome/scalable/status/mail-unread.svg
      icon=`bash ~/scripts/twitter-user-pic $sender`
      content=${message[@]:1}
      notify-send -i $icon -t 15000 "$sender" "$content"
      for terminal in `ls /dev/pts`
      do
        echo "$sender $content" /dev/pts/$terminal
      done
    done
  else
    echo "Nothing new."
  fi
  sleep 60
done

The "twitter-user-pic" bash script downloads and stores user's profile images. It looks like this.

#!/bin/bash
 
if [ ! -z "$1" ]
then
  picture="$HOME/graphics/avatars/twitter/$1"
  if [ -f "$picture" ]
  then
    echo $picture
  else
    url=`python2.6 ~/scripts/python/twitchy/twitchy.py picture -u "$1"`
    if [ ! -z "$url" ]
    then
      picture="$HOME/graphics/avatars/twitter/$1"
 
      wget -qO - "$url"  "$picture"
 
      echo $picture
    fi
  fi
fi

"twitchy.py" is my own little Python script that can currently only query user profile picture URLs (may be extended later). As you can see, the twitter-user-pic script checks a local file cache to see if the picture exists, and otherwise downloads it. Everything about this is rickety and hacky, starting from the lack of file extensions.

This is my python script:

#!/usr/bin/python2.6
import sys;
import getopt;
import twitter;
 
def main(argv):
  api = twitter.Api();
  api.SetCredentials("arancaytar", "OC3TaQHqONi1PuAr");
  
  command = ""
  user = ""
 
  if argv[0][0] != "-":
    command = argv[0]
    argv = argv[1:]
  opts, args = getopt.getopt(argv, "c:u:", ["command=", "user="]);
  for opt, arg in opts:
    if opt in ("-c", "--command"):
      command = arg
    elif opt in ("-u", "--user"):
      user = arg
  if user != "":
    account = api.GetUser(user)
    if command == "picture":
      print account.profile_image_url
  else:
    if command == "picture":
      print "picture command requires -u user"
 
 
main(sys.argv[1:])

So now you can see that while the endless-loop script runs, I will get messages sent both to the desktop (where they are unfortunately not clickable, as notify-send doesn't do actions). Instead, I'm broadcasting them (again, in a pretty dirty way, using /dev/pts/*) to all open TTYs (which I'm anticipating will crash something badly, but nothing has happened so far).

Then I have a starting and stopping script that works like this:

#!/bin/bash
if [ "$1" == 'start' ]
then
  if [ ! -f "$HOME/.twitterd.pid" ]
  then
    echo "Starting"
    ~/scripts/twitcron.sh /dev/null 2/dev/null 
    echo $!  "$HOME/.twitterd.pid"
  else
    echo "Already running: "`cat $HOME/.twitterd.pid`
  fi
else
  if [ "$1" == 'stop' ]
  then
    if [ -f "$HOME/.twitterd.pid" ]
    then
      echo "Stopping"
      pid=`cat $HOME/.twitterd.pid`
      kill $pid
      rm $HOME/.twitterd.pid
    else
      echo "Not running."
    fi
  fi
fi

And it actually works! I've launched the process and now get those tweets live to the desktop and the terminal. It's great.

Half a year has passed since the last theme change, so tradition dictates I pick a new one again. Actually, this time I wanted to change because Agregado lacks fluid-width support. Seeing your website on a 2000px monitor, all alone in a narrow vertical beam in the center, really makes you question your web skills.

This new one is Colourised, and it is lovely. Lovely I say. I particularly love the prismatic spectrum in the background.

Naturally, I had to tweak quite heavily, as with all themes. The default is for the content to start well in the bottom half of the screen, for no other clear reason than to offer a clear view of the background image. Which is admittedly lovely, but it can be admired just as well behind the text. My tweak shifts the header, slogan and content all up a few notches.

Afte setting up the theme, my next priority was getting my page to validate again. This is something you need to retest regularly, because there will always be something around to break it again. In this case, I had to rewrite Drupal's entire linebreak generator, which wasn't able to work around in-line divs and headings in XHTML-valid ways. Pleased to say that it works very well now.

You can also see I have added a Latest Tweets block to my sidebar, since I'm using Twitter quite heavily these days.

I think I've been coding on one thing or another (proxydb, dhtml_menu, shell scripts, xbbcode) for well over twelve hours now. Mostly to avoid having to follow the news. I don't think I've felt physically ill like this since March 2003. Why do politics need to turn wonderful human beings into martyrs?

I've toiled on a few more features for the Proxy DB module (described in the past few posts).

Changes include:

• Password protection is optional (if no password is set, the list of random proxies is freely displayed).
• If the PHP installation supports GeoIP, and a list of flag icons is independently downloaded to flag-icons/ in the file directory, then the location of every proxy will be marked with a flag icon.
• Whole lists of proxies can now be submitted in batch (by uploading a file or pasting its contents directly). These lists can include proxies with multiple ports open (such as 3128,8080-8090,55555). The lot will be tested using Drupal's Batch API, with a neat progress bar, and only functional proxies will be saved.

The Proxy DB module is available for download from the Ermarian Network, and there is a live production site at barred.ermarian.net.

* * *

In other news, it appears a young woman was killed in the protests (among many others) yesterday. RIP Neda.

My last post on a Drupal-based database of proxy servers provided a link to the proxydb module I wrote. However, realistically the only potential user of the module, right now, is myself, since it is a very buggy unfinished version. So I set up a site ready for production use (after much further debugging).

The site runs on Drupal 7, which is extremely sleek. I still get almost 9M memory peak for bootstrap, unfortunately - but premature performance-tweaking is the root of all evil.

The newest code of the module can be downloaded at proxydb-7.x-0.2-r355.tar.gz.

The production site is at barred.ermarian.net. (I had the barred subdomain left over, and it seemed close enough in meaning to be repurposed for this).

---

Note that I am a newcomer to all this: Austin Heap already has a very good proxy list running (the development of which I'm following, and which I might contribute to as well). However, just as the proxies themselves, these resources are all at risk of filtering, so you could say "the more the merrier".

After my last post on testing proxy servers, I spent the past 10 or so hours on what inevitably happens whenever I develop even the simplest web application: A Drupal module.

This particular module incorporates your basic "database table, create record, administer records" functionality for web proxies, but also uses a testing function similar to the one in the previous post, as well as a "continuous retesting" feature that rechecks the 15 oldest servers on each cron run (and eventually gives up on a server after a test repeatedly fails).

It's still full of bugs, since I don't actually have a database with dozens of proxies, and some parts are still untested. But I've managed to run it locally, add my own local proxy server to it, and then see it in the list.

The code is available over at the Ermarian Network, like my other Drupal modules that are not fit for d.o contrib (yet).

(Disclaimer: The above does not endorse any political view beyond that of defeating censorship in all its forms. As the sadly-fictional UN comissioner Pravin Lal said in Alpha Centauri: "Beware of anyone who would deny you access to information." I'll interfere in that any day. )

This is a simple tool to check a list of proxy servers for uptime. The scripts are available in zipped form (proxydb.tar.gz). You can also check out the SVN repository:

Quoting from the README file:

Requirements

• Ability to run PHP scripts in command prompt.
• PHP with curl extension.

Setup

Enter proxy files to be checked into a text file like this:

Edit main.php to enter the text file's name into the constant PROXYDB_FILE_PROXIES.

Usage

The script reads from standard input and writes to standard output by default.

If a command-line argument is passed, that will be considered the filename of the input file.

The second column of the tab-separated output will be either "up" or "down". The third is either "0" (if the IP did not respond) or a 3-digit HTTP response code. Only response codes of 200 are currently considered successful.

Configuration:

Open config.php with a text editor and change the constants defined in it. The available config values are as follows:

CONFIG_PROXYDB_FILE_INPUT [ = "-" ]

Input file. Can be overridden by the first command-line arg. A value of "-" refers to standard input.

CONFIG_PROXYDB_FILE_OUTPUT [ = "-" ]

Output file. A value of "-" refers to standard output.

CONFIG_PROXYDB_CHECK_URL

The URL that should be queried for testing. This URL must return a response of 200 (not, for example, 302 or 301). By default, the script downloads the Google Logo, being a small highly available static file on a network with good bandwidth.

CONFIG_PROXYDB_CHECK_TIMEOUT [ = 15 ]

The number of seconds to allow for a request to finish, in total.