Command Line Access to OS X Filesystem Meta Data

Apple’s OS X is cool.

It occurred to me today to wonder what features of Spotlight could be accessed through the command line (terminal). After some poking around on the Internet I came across some documentation for the “mdls” and “mdfind” commands.

The “md” in the file names stands for “meta data”. For those of you who are unaware, meta data is data about data or data describing data. All files on your computer have meta data, stuff like the date of creation of the file or the file type or size. Apple makes meta data very rich within OS X and has even made file contents part of the searchable meta data as well, which is cool.

Here’s the documentation on mdls and mdfind.

So what sort of things become possible with this sort of functionality? Lots of super geeky things, it turns out. More practically, consider that, with a “cron” job creating a nightly archive of all recently modified files, a script could then be written to “tar” those files and ftp them to a web server. My Dreamhost account gives me 200GB of storage, so I should be good to go for a while since my Mac only has an 80GB HDD. Here’s what the bash script to get the file listing and tar the files would look like:

#!/bin/bash
MYDATE=$(date +%Y%m%d)
mdfind '(kMDItemContentModificationDate > $time.today)' >archive.txt
tar cf archive_$MYDATE.tar -T archive.txt
perl send2ftp.pl archive_$MYDATE.tar

The Perl script that is called in the end of the bash script would look something like this:

use Net::FTP;
$ftpobj = Net::FTP -> new ("mysite.com");
$ftpobj -> login("user","password");
$ftpobj -> cwd ("/archives");
$ftpobj -> put ($ARGV[0]);
$ftpobj -> quit;

If this script were run daily at a few minutes before midnight, it would capture everything done in a day and send it to the web server. That’s cool.

I’m Not Afraid to Admit it When I’m Wrong

But am I?

I wrote earlier about some misgivings I had regarding Apple’s (at that time rumored) move into the cell phone market. The post was melodramatically titled, “iPhone - The Beginning of the End for Apple.” Apple ignored me and introduced the iPhone on Tuesday at Macworld San Francisco - it certainly is an impressive device. Steve Jobs did some of his best work Tuesday in selling the iPhone but there are a few things I have reservations about still.

Such as, who is the iPhone for? Not for kids. Or rather, not for kids who aren’t spoiled rotten by parents desperate for love from their ungrateful hellspawn. Starting at $499, the iPhone has the features and the price to compete directly against the premier “smart” phone companies and products. We’re talking Blackberry, Treo, Blackjack, etc. These guys have years in the market and have very competitive products - we can expect that they will compete very vigorously with Apple, not just roll over and take their beating.

NBEHTM addresses some of my concerns by asserting that the introduction of the iPhone is not a ploy by Jobs to take on the cell phone market but rather a ploy to increase the penetration of Apple computer hardware into the enterprise. This makes some sense given that business is where most of the Blackberries are going, and given that Apple’s presence in the business community is pretty weak. An iPhone could extend the so-called “iPod aura/halo” into the boardroom. My one complaint with this idea is that Apple hasn’t shown a 6G iPod as yet, so either they’ve got another product up their sleeves or they’re going to be trying to direct customers toward an expensive phone/media player hybrid. Jobs wants a 1% piece of the phone market, and he’ll likely get it, but how will that affect the sales of devices that can be truly called iPods if a upper-middle class iPod option isn’t on the market? And will the iPhone fit with corporate purchasing plans? And do executives really want to learn something new, or would they rather stick with what they know?

Whatever the corporate response to the iPhone, Apple isn’t dense about its media players and has a version of the full-screen iPod, which will look remarkably like the iPhone, ready to go.

Anecdotally, I showed the iPhone to Mrs. Interested (http://www.apple.com/iphone/phone). She went from “$499 is a lot of money for a phone” to “$499 isn’t that much money for a phone” really quickly. Time will tell what happens to Apple and the iPhone - I’m looking forward to seeing the situation unfold.

Apple’s Automator is the Gift That Keeps on Giving

Automator is a built in feature of Apple OS X, starting with version 10.4. I’ve used it in a variety of situations and the more I use it, the more I find I really enjoy it.

In a nutshell, Automator is a graphical way to script/automate many of the features of your Apple computer. Want to resize and rename a large number of pictures? Easily done with Automator, but it’s also possible to apply color correction and other effects en mass as well. Want to copy all your email to your iPod in the morning so that you can read it on the subway to work in the morning? Again, easy. Here’s what an Automator workflow looks like:

Automator Workflow Example

This is cool and, incidentally, a feature that Microsoft hasn’t even tried to emulate in Vista.

Automator workflows can be run through the graphical interface but I just realized that they can also be run from the command line. The workflow icon that is saved to your Desktop (or wherever) looks like a file but is in reality a directory. Hidden under the path example.workflow/Content/ is a file called document.wflow. The document.wflow file can be invoked by Automator from the command line as shown below:

Automator Invoked from Command Line

What this means is that the general Automator coolness can further extended by scheduling Automator to run workflows without any user intervention. The scheduling service we use to do this is called “cron”, a UNIX utility that comes pre-installed on Mac by virtue of the BSD core of OS X. There are a variety of circumstances where scheduling workflows could be useful. Using the email-to-ipod-copying example from above, you can schedule your Automator workflow to copy emails to your iPod 10 minutes before you usually walk out the door in the morning so that all you need to do is grab and go. In order to set up a scheduled Automator workflow, do the following:

  1. Set up a workflow and save it. For this example we’ll assume that the workflow is named “example” and it is being saved to the Desktop. Your Automator workflow will look like this:

    Automator Icon

    but remember that it’s in fact a directory and not a file.

  2. Create a text file that will be hold the schedule information we will feed to “cron”. We’re going to be using the terminal in just a few minutes to start the cron job, so might as well get warmed up. Change directories to Desktop (’cd Desktop/’) and run the “nano” editor by using the following command:

    Command to invoke Nano

    Assuming we want our workflow to run once a day at 7:30 in the morning, the file would look like this:

    crontab file

  3. Next we load the job into our “crontab”. This is illustrated below, along with how to check to make sure everything made it in alright:

    loading and checking crontab

So that’s it. There’s a lot more that can be done here - for example you could set up the cron job to run only on weekdays by modifying the cron job slightly, but that’s probably out of the scope of this post.

What do you all think? Have you got interesting or inventive uses of Automator that you’d like to share?

Right Place, Right Time

The story of Larry Page and Sergey Brin is like that of many other successful people - they were at the right place at the right time.

The thing that jumped out at me is the number of times things could have worked against the two of them. Persistence and dumb luck had a lot to do with where they are today (not everything, but a lot). I want to be that dumb/lucky.

Adventures in Web Hosting

The RealMetrics blog posed the question: Could Your Web Host be Missing Their Uptime SLA 37% of the Time?. They reviewed 50 web hosts over six months, measuring both availability and responsiveness and posted their results. Unfortunately they didn’t have anything from Dreamhost, the host behind this site.

There are many difficulties in measuring things like shared hosting. A positive experience with a host might be the result of chance placement with inactive accounts on the same server or could be negative for the opposite reason. I’ve had both good and bad experiences with Dreamhost since I started with them. The uptime that they’ve offered has been pretty good except for when they died right during two minor traffic surges I had a couple months back - just when you’d like your host to be up, it turns out.

In fact, those server deaths infuriated me enough to start my own uptime monitoring. My records haven’t been kept very assiduously, and it appears that since I started doing pings and regular downloads of content for test from my sites Dreamhost has been nothing if not stable.

For the curious, I just set up a super simple Perl script to ping the server every minute and output unavailability to a text file. Another script downloads a file at regular intervals throughout the day to gauge the average speed of the server.

Perl ping script:

#!/usr/bin/perl
use Net::Ping;
$| = 1; # sets output to unbuffered
$p = Net::Ping->new();
$host = $ARGV[0]; # takes the web host as single argument
$sequence = 0;
while ()
{
sleep(60); # run every minute
$sequence++;
next if $p->ping($host); # don't report if things are fine
my $time = localtime();
print $sequence . "\t dead \t" . $time . "\n";
}
$p->close();

Running the above code in the command line as follows:

./ping.pl whateversinteresting.com > pings.txt

writes errors to the file pings.txt (”pings” can be monitored in another terminal window with the “tail -f pings.txt” command).

I’ll add the downloads script later.

I Want to Live in Space

The early 1970s were an interesting time for NASA. Having achieved the epic goal of men on the moon - and in so doing shattering the perception of Russian dominance of space - the people at NASA needed to scramble to develop a new vision for the space program in order to maintain the publics fickle attentions. I wrote about this previously here.

The vision that NASA developed was nothing if not grand. It involved the colonization of space on a scale so enormous that few in science fiction have managed to surpass it. I remember reading books when I was a lad (not too long back) with some of the following pictures in them and thinking to myself that I’d like to live in space. Looking at the pictures now I still would like to live in space if living in space bore any semblance to what NASA was able to dream up.

All pictures are from NASA - the actual NASA page includes sixteen full color illustrations but ten are shown here:

Cylindrical Colony External View

The external view of a double cylinder colony.

Cylindrical Colony Internal View

Internal view of cylindrical colony. Arthur C. Clarke’s “Rama” included a cylindrical space ship much in this configuration.

Cylindrical Colony Internal View

Another view of the interior of the cylindrical colony. Note the suspension bridge crossing the body of water.

Variations on a Toroidal colony. This is probably the most familiar type of space station and was featured prominently in movies such as “2001 - A Space Odyssey.”

One thought that was common in the 1970’s was that aerable land would become more and more scarce on Earth, leading to the need for agriculture in space. Large mirrors would reflect light into the interior of the habitat, making possible large scale growing of food.

The 70s architecture is killer.

Exterior View of Bernal Sphere

This is an exterior view of a Bernal Sphere. The space stations referred to in William Gibson’s Neuromancer and the space station Babylon 5 are Bernal Spheres.

Interior View of Bernal Sphere

Interior View of Bernal Sphere

Interior views of a Bernal Sphere. Note the suburban lifestyle the pictures seem to advocate and portray. Also note the “human-powered” flying device in the second picture. A spinning sphere doesn’t produce actual gravity so the things such as a flying bike-plane could become possible. Cool.

Building the sphere.

Am I the only one who thinks this future in space looks enticing? I am fully aware of how big a deal it would be to kick something like what is shown above into gear, but I think it would be worth it.

Open Source at Work

Nothing aids in understanding better than application - learning “theory” doesn’t come close. In a previous post I wrote, theoretically, about how great Open Source is. This post shows the proof.

In the other post I pointed out that I had enabled a “share this” feature on this site to enable posting of Whatever’s Interesting posts to social news sites like Digg (regardless of past issues I’ve expressed concerning Digg). I casually pointed out that there were other options to the “Share This” plugin, written by Alex King, and I chose and linked to “Digg This Reloaded” by Ankur Kothari in specific.

Not ten hours after I posted, Ankur had written a comment to point out that DTR now actually includes Alex King’s plugin, while including more features - in specific an options page in the site admin and other nifty things that Share This lacked. Awesome, and so now Wi is running DTR.

And while this shows how Open Source can work toward a positive result technically, I think the real point to make here is something I neglected to mention the first time around. Open Source software is great from a technical perspective but the people behind Open Source are greater. The actual developer. in this instance, took the time to clarify a point on a random blog, advocating for his work. How many developers at Microsoft go home at the end of the day and try to set the world straight about random features of Windows?

Maybe some do, I don’t know, but the point is that there is real enthusiasm and love for software and community in Open Source. And that’s why Open Source will win in the end.

Aliens are People Too

Far SideI love the quote, “The surest sign of alien intelligence in the galaxy is that they aren’t here.”

A big question that has puzzled scientists is that, if there is extraterrestrial life out there in the galaxy, why can’t we find it? The assertion in this post is that they’re all at home watching “Alien Idol”.

Whether or not you believe in aliens, this observation is brilliant as a veiled commentary on humanity itself. Why aren’t we half way to the stars by now? Why are we waiting for the aliens to come to us? I’ve asked the question before in the context of space tourism - is that the noblest good that our space program can result in?

Our present society is making some very interesting choices for itself. By interesting I mean not good. I think it’s dangerous to accept limitations such as are placed on us by space travel. Not doing things because they are hard is a dangerous precedent to become comfortable with. No culture on earth survived long because it made choices to avoid difficulty. We do so now at our own peril.