The Molecular Universe

I have been surprised sometimes to see how much disk space empty directories consume. I suppose that this depends on the file system that you are using, but nevertheless sometimes it is convenient to find and potentially remove empty directories. (I seem to find that empty directories on Windows are particularly disk space usage heavy).

Here is a script which accomplishes this using the find command. I am including the author's name - I think I tweaked the script slightly at some point to report what it was going to do, rather than actually doing it. If you want it to remove empty directories change the DOIT variable to "y". Actually the rmdir command will only remove an actually empty directory, so the script is fairly safe. However, use at your own risk, as always.

#!/bin/sh -f
# rmemptydir - remove empty directories
# Heiner Steven (heiner.steven@odn.de), 2000-07-17
#
# Category: File Utilities

DOIT="n"

[ $# -lt 1 ] && set -- .

find "$@" -depth -type d -print |
    while read dir
    do
        [ `ls -a "$dir" | wc -l` -lt 3 ] || continue
        echo >&2 "$0: removing empty directory: $dir"
        if [ $DOIT = "y" ]
        then
          rmdir "$dir" || exit $?
        fi
    done
exit 0

The Rational Project Manager

The Rational Manager is a management classic - The authors are from the management consultancy founded by Kepner and Tregoe, and the company influence on this book on project management is clear. Kepner-Tregoe strengths are decision making and problem solving and naturally this volume focuses on these aspects of project management. In addition to the consistently Kepner-Tregoe tone the book contains a logical breakdown of project management through the following overall flow: loosely federated project teams are now common, project definition is critical, planning, and assigning responsibility, implementing, managing people, decision making, and problem solving. A final chapter covers the possibility of installing project management software organization-wide. The people management and problem solving chapters are the most useful, in my view. Many modern project managers are faced with the problem of working with teams where the members have diverse reporting responsibilities and often the needs of the organization are a complex blend of conflicting priorities. In these circumstances the project manager needs to be careful - and diplomatic - and the methodologies in these chapters are useful in the negotiations, upon which the success of many projects now hinge.

Behind Closed Doors

I bought this book after listening to an interview with Johanna Rothman in a podcast. My overall conclusion having read the book is that - although it is interesting - it does not address any of the problems that I see in software management. I am unsure where this 'miss' lies. Perhaps I failed to approach the book with an open enough mind to pick up on its critical insights or understand how the portrayed examples applied to my own challenges. I was suspicious of the superficial treatment given to the characters (the book portrays a star performer manager through his interactions with his team) and I was suspicious of the short term-ism in the depicted solutions, be they agile or otherwise. Long term planning and career development are not addressed by the book - the approach is immediate and oriented toward quick fixes for troubled (fictional?) teams. The lack of strategy worried me throughout. The essential problem in my value extraction from the book lay in three areas: the issues that the fictional characters encounter are not particularly serious - technology choices, no hiring, no firing, no restructuring nor acquisitions; the book is light on its portrayal of technology - addressing this would have helped build credibility in its anecdotes; and the checklists and forms included are also light and not particularly new.

If your web site has a page of external links it is useful to have an automated method to check that the links are valid, because you cannot tell when a link will change or be taken over by another site. The alternative to automation is a tedious session of pointing, clicking and 'back'-ing. Of course, there are some heavy duty link checking programs which can automate the task for you - but even they tend to be a little tedious to use, meaning that your link checking might never get done, and your site would start to exhibit decaying links, a sure sign of neglect and carelessness. The following script assumes that you have a page called 'links.txt' which contains your external links. It then processes the strings which begin with http within this page - to pull down just a single page from each site, which it stores as www.site.com.tmp in the current directory. If a site is off line, or a page has moved on a given site, then you will see evidence of these facts in the output of the script. Rather than use 'lynx' or 'curl' to download the target page, the scripts uses 'telnet' and individual requests to the http server. This is done because it is a lot more educational than simply using 'lynx' or 'curl' and having some understanding of http is a good thing! The script took some inspiration from ancient variants which have existed on the web for more than 25 years - but is modified to not use temporary files. If you have any problems with it - please don't hesitate to let me know. At some point, I may put into the script a retry if the current sleep times prove to be ineffective in producing reliable page downloads. However, in its current form the script seems to work fine for my links.txt page - it gives me the confidence to extend the page knowing that I will be able to keep it up to date despite the ever changing web. Here is the script.

#!/bin/sh

while read siteline
do
  nohttp=`echo $siteline|sed 's|http://||g'`
  site=`echo $nohttp|sed 's|/.*$||g'`
  item=`echo $nohttp|sed 's|/| |'  | awk '{ print "/"$2 }'`
  echo Checking $site $item
  (echo "open $site 80"; sleep 3; echo "GET $item HTTP/1.0"; \
   echo -n "User-Agent: Mozilla/5.0 "; \
   echo -n "(Windows; U; Windows NT 5.1; en-US; rv:1.8.1.4)"; \
   echo " Gecko/20070515 Firefox/2.0.0.4"; \
   echo "Host: $site"; echo; echo; sleep 5) | telnet > telnet.tmp
  grep "404" telnet.tmp | grep "Not Found"
  cp telnet.tmp $site.tmp
  echo "This site produced" `wc -c telnet.tmp | awk '{print $1}'` "bytes"
done < links.txt

The Art of Project Management

The Art of Project Management is dedicated to software project management. Written by an ex-project manager from Microsoft, Scott Berkun, this is an interesting and valuable book. Interesting because of the wealth of experience with which Scott delivers in the book. There are anecdotes, characters and sketches from the corridors of Microsoft on every other page. Scott was at Microsoft during an exciting time - and the projects and decisions that he made have had a substantial impact on the world of software - so it is fascinating to learn something of that experience. However, in addition to the anecdotes, the book provide considerable practical and valuable advice. The book is structured around three sections: plans - which discusses visions and estimates; skills which focuses on software specifications and communication techniques; and finally management and dealing with the organizational side of software projects. Almost every page provided interest and value to me - and I thoroughly recommend the book. Microsoft in the 1990s contained some of my favorite authors: Richard Brodie, Joel Spolsky and Scott Berkun. I wonder if it was fun or stressful work there?! Scott's web site is also worth a visit: http://www.scottberkun.com.

Here is a short script which goes through a directory structure and operates on mp3 or MP3 files. It uses standard bash methods - and I am posting it here in case you are interested - of course you can achieve the same effect with:

find . -name "*.[mM][pP]3" -print

...but a script is more fun - and substantially slower - giving you time to think about other things - which is often useful. The script prints out the names of the files it encounters - you could have it do something else - like report checksums (cksum) or modification times (ls -lt). The necessary lines for these operations have been commented out in the script. You can use this script as a template to carry out other file based scripted operations. Also note that it won't find files with the following cases in their extensions: mP3 or Mp3 (i.e. mixed case file names).

#!/bin/bash
scandir () {
for item in *
do
  if [ -f "$item" ] ; then
    curdir=`pwd`
    nfile=`expr $nfile + 1`
    EXT=`echo "$item" | rev | cut -c 1-3 | rev`
    if [ $EXT = "mp3" -o $EXT = "MP3" ]
    then
      echo "$curdir"/"$item" | cut -c${ld1}-
#      ls -lt "$curdir"/"$item"
#      cksum "$curdir"/"$item"
    fi
  elif [ -d "$item" ] ; then
    cd "$item"
    scandir
    ndirs=`expr $ndirs + 1`
    cd ..
  fi
done
}
startdir=`pwd`
ld1=`echo $startdir | wc -c`
ld1=`expr $ld1 + 1`
echo "Initial directory = $startdir"
ndirs=0
nfile=0
scandir
echo "Total directories searched = $ndirs"
echo "Total files = $nfile"

With Windows you tend to add more and more programs and services during the lifetime of the computer, as you try to get more and more pieces of hardware and assorted technologies up and running on the system. Each new layer of software encrustation tends to stay - because after a while you can no longer tell if you actually need a given program. Virus scanners, firewalls, printer monitors, software that looks after your sister's scanner(s), it all adds up - and after a while you notice that the machine is not as fast as it used to be. Well, to add a little science to the exploration of what is causing the most significant problems here is a great link: http://www.thepcspy.com/read/what_really_slows_windows_down. If you follow the link you will find a detailed analysis of different virus scanning programs and their effects on boot time, computational performance and disk access times. The results are very interesting and the methodology can be applied to any programs or processes that you might find questionable on your own machine. Of particular interest is a simple timing program which checks the arithmetic performance of your CPU and times disk access on your machine. This is a recommended article for anyone with an interest in keeping their machines running at a reasonable rate.

Virus of the Mind

Having thoroughly enjoyed Getting Past Ok, naturally I had to read Virus of the Mind. And I was not disappointed - it is another very interesting read. The discussion of memes around the web and world has been substantial. However, personally, I had not thought about memes too much until this read. (My theory was that fashions are well understood and have existed for thousands of years - there was probably nothing new in memes). Although, this book is titled 'new science' and occasionally comments that memetics is a 'new science', I managed to be able resist that notion without without resisting the general ideas that the book presents and summarizes. The chief idea is: - that the transmissibility of ideas is a primary property which determines the success of an idea. Transmissibility is not equated with usefulness or other forms of value. Ideas exploit the wiring needed by humans to survive and thrive. We are overly interested in danger, competition for resources, nutrition and expanding the tribe.

The four Fs: Fear, Fighting, Food and 'Finding Mates' - are hardwired into us and lead us to be particularly attuned to ideas which combine these interests. We have additional sensitivities, such as a reluctance to avoid cheaps insurance and bargains in general, which make life easier for idea and meme manufacturers. I was fascinated by the book - and I searched the web for additional information on memes.

This produces a torrent of information, of course. However, here are two useful meme related links: http://www.memecentral.com - this is Richard Brodies' web site dedicated to memes. I believe that it is not as actively maintained as it once was - but it is an interesting resource. There is an mp3 of an interview with Richard Brodie from 1996, on Radio Curious. It is well worth a listen.

The Psychology of Persuasion

The Psychology of Persuasion: How to Persuade Others to Your Way of Thinking by Kevin Hogan

The Psychology of Persuasion - is aimed at people in marketing and sales. It provides familiar information - if you read the types of books which are reviewed on FictionFobic - and found in the self-help sections of bookshops. Kevin Hogan is an expert on body language and a speaker - and contributes to a variety of organizations that consume information that supports the need to understand and connect with customers and audiences. So, to me, much of the book read like a speech - the examples were present in large numbers but there was not much detail or solid proof behind the examples. On NLP techniques - particularly the possibly hard to believe bits about eye movements and peoples' thoughts I am inclined to believe the explanations given by Nick Boothman and Greg Hartley. However, the Psychology of Persuasion is a good summary. Again it is interesting to note that there is title duplication afoot. Cialdini's 'Persuasion' is sub-titled 'The Psychology of Persuasion'. It is almost as though there is a deliberate duplication of the title words sometimes applied by publishers to interesting themes(?). (I resisted the temptation to write 'memes').

Here is a simple Bash script which illustrates the use of find and awk in determining which files to update when synchronizing two directories. You could use the rsync command to carry out this task for you. However, rsync is a little too efficient and terse, and an open script, even if it is mainly awk, allows you to understand precisely what is about to happen to your files. The script works by using 'find' to gather data about the directories being synchronized. A list of synchronization commands (simple cp's or rm's) are presented to the user based on this analysis, and the user can then decide whether to execute the commands or not. In my tests the analysis of 1.3 GB of files on hard drive and on USB took around 3 seconds - so the speed of this script is not far from the speed of rsync itself.

The usage reporting and error checking are minimal, the arguments are: ds.sh directory1 directory2 time-window (in seconds).

(and all arguments are compulsory). So, for example, you might type:

./ds.sh usb_documents drive_documents 2

to synchronize your USB drive documents with your hard drive documents.

This script is experimental - please feel to use this - at your own risk. If you have questions or comments please let me know.

#!/bin/sh
awk '
BEGIN{
  timewindow=ARGV[3]
  print "The time window is: " timewindow
  dir1="\"" ARGV[1] "\"/"
  dir2="\"" ARGV[2] "\"/"
  readdir(dir1, lista, typea)
  readdir(dir2, listb, typeb)
  for(filea in lista){
    if(filea in listb){
      if(typea[filea] == "f"){
        timediff=lista[filea]-listb[filea]
        if(timediff > timewindow){
          com[++ncom]="# file in source directory newer than target"
          com[++ncom]="cp -a " dir1 "\"" filea "\"" " " dir2 "\"" filea "\""
        }
        if(timediff < -timewindow){
          print "# WARNING NEWER FILE IN TARGET DIRECTORY"
          print "# files concerned are: "
          print "# " dir1 "\"" filea "\"" " " dir2 "\"" filea "\""
        }
      }
    }else{
      if(typea[filea] == "d" ){
        dcom[++ndcom]="# directory needs to be created in the target"
        dcom[++ndcom]="mkdir -p " substr(dir2,1,length(dir2)-2) \
                       substr(filea,2) "\""
      } else {
        com[++ncom]="# file needs to be copied to the target"
        com[++ncom]="cp -a " substr(dir1,1,length(dir1)-2) substr(filea,2) \
                 "\"" " " substr(dir2,1,length(dir2)-2) substr(filea,2) "\""
      }
    }
  }
  for(fileb in listb){
    if(!(fileb in lista)){
      com[++ncom]="# need to remove file in target not in source"
      com[++ncom]="rm -f " dir2 "\"" fileb "\""
    }
  }
  if(!ncom){
    print "No updates required"
    exit
  }
  print "The following commands are needed to synchronize directories:"
  for(i=1;i<=ndcom;i++){
    print dcom[i]
  }
  for(i=1;i<=ncom;i++){
    print com[i]
  }
  print "Do you want to execute these commands?"
  getline ans < "/dev/tty"
  if( ans == "y" || ans == "Y"){
    for(i=1;i<=ndcom;i++){
      print "Executing: " dcom[i]
      escapefilename(dcom[i])
      system(dcom[i])
      close (dcom[i])
    }
    for(i=1;i<=ncom;i++){
      print "Executing: " com[i]
      escapefilename(com[i])
      system(com[i])
      close (com[i])
    }
  }
}
function escapefilename(name){
  gsub("\\$", "\\$", name)     # deal with dollars in filename
  gsub("\\(", "\\(", name)     # and parentheses
  gsub("\\)", "\\)", name)
}
function readdir(dir, list, type,        timestamp, ftype, name){
  cmd="cd " dir ";find . -printf \"%T@\\t%y\\t%p\\n\""
  print "Building list of files in: " dir
  while (cmd | getline > 0){
    timestamp=$1
    ftype=$2
    $1=$2=""
    name=substr($0,3)
    list[name]=int(timestamp)
    type[name]=ftype
  }
  close(cmd)
}' "$1" "$2" "$3"

Ready for Anything

'Ready for Anything' is a set of essays written by David Allen, originally designed to be distributed to email subscribers. After the success of Getting Things Done, and undoubtedly with the enthusiastic support of the publisher, Viking, the collected 52 essay-messages were bound and sold in their own right. The result is a nicely written collection of sound-bites on the GTD philosophy (as Getting Things Done has become known) and David Allen's productivity system in general. Given the diminished attention span of the age - from which we all now suffer - and the diverse demands on our time and energy - this topic has a universal and enduring appeal.

The 'Ready for Anything' essays do not disappoint. David's easy phrasing is evident in most - and the message is uncomplicated and reassuring. As far as I can determine it can be summed up in a television or radio interview format as: collect, process, organize, review and do. And indeed essay number 5 provides precisely this summary. David recommends focusing on the next action and clearing the decks and mind. Great advice, of course, because starting with mission statements and values create quite a barrier to further action. It isn't obvious to me why there is such an industry associated with GTD - with many sites, software and resources associated with the GTD-meme. However, it does not seem to be in any way harmful and is clearly strongly transmissible. The essence of time management is decision making. At any given moment - do you surf the web, watch YouTube, or do some work? (There are really only 3 choices). GTD provides a decision framework - and with GTD - you can almost convince yourself that you can do all three at once. This easy relevance and possible alignment with the puritan work ethic may account for some of the popularity of matters time management, and personal-productivity in general.