Main

February 2, 2010

New book from a collague: "Pull: The Power of the Semantic Web"

David Siegel, a former colleague from a previous venture, has a new book out on what might be called "Web 3.0" or "The Semantic Web". It's called Pull: The Power of the Semantic Web to Transform Your Business , and it hearkens back to David's blog The Power of Pull. David's an interesting guy and always has something thought-provoking to say. I haven't read the book yet, but I have read a lot of his writings that led up to this book and his blog and found them stimulating. DISCLAIMER: I don't get anything other than good vibes back from David for mentioning his book. Not even a free book.

January 1, 2010

"All these worlds are yours except Europa."

So was the enigmatic message from HAL 9000 in Arthur C. Clarke's 2010: Odyssey Two...

And so now we start a fresh new decade. It seems to be an American tradition to come up with resolutions for the New Year to do things (we Jews have been doing this for our New Year celebrations, which tend to be much more sober affairs than secular New Years, for generations.), and so as not to break with this tradition, I offer mine own insights and hopes for the new year, both personal and professional.

Personally, I'm hoping to achieve a number of things, including finishing off the last 20 lbs or so of weight loss I've been working on (I'm about halfway there -- ideally I'd like to be under 200 lbs, which is a place I've not been since I was about 17 years old) and I have incentives this year in that I'm hoping to test for Black Belt in my karate organization (which does not officially require me to lose the weight, but sure, it has held be back and continues to hold me back.) I'm reasonably confident that I'll be able to do it "this year for sure", since coming into 2009 I had no idea that I'd ever be able to run 5 kilometers, and I managed to hit that a few times before I damaged my achilles tendon during the summer...

Professionally, I've got a number of goals, among them:

  1. Updating my company's website 3 Phase Computing -- that link points to the old site, not the new one
  2. I, personally, have never really been into marketing, but this has to change now, as 3 Phase has to grow, so I've decided to begin marketing in a truly geek fashion: by writing papers and software to establish a name, rather than make vague promises backed up by smoke and mirrors. So this part is to actually write papers/blog entries (not here) and post them to the corporate website. I'm also counting on the abilities of my talented co-workers as well.
  3. I have a Ph.D. already, so in some cases having a long string of professional certifications in addition may label me as more of a "test-taker" than a "do-er". Nonetheless, just for my own edification, so I know that I know some material, I'm sitting a few of these exams. If it turns out that presenting these additional certifications helps with marketing, then all the better, but I'm not counting on it.
  4. Finally, although I don't spend much of my time writing code any more, my resolution in this regard is to write better code so I don't need to purchase offsets for my ... code. Luckily, this is a relatively easy resolution to keep, because I have so many other hands who are looking at my code and complaining to me about it.
  5. There is a long list of new technologies that I've not sufficiently explored or played with:
    • ZFS and Solaris Containers, and upgrading my develoment environment to FreeBSD 8.
    • While I've used VMWare quite a bit in the past, trying some of the new, fancy features in the enterprise suite are things that I need to definitely get my hands on more frequently,
    • Some other smaller projects, like network shell
    • Larger, new programming areas, like Python 3
    • And of course, there is the whole new NoSQL/Document database burgeoning movement that has captured so much mindshare and intellectual energy recently.

Continue reading ""All these worlds are yours except Europa."" »

December 7, 2009

Attention, recruiters: please don't lie to us.

This is an expansion on the 140 characters I spoke about on Twitter a little while ago. This is as much of the conversation as I can remember.

I just received a call from an "IT recruiter" (I don't remember who, and it doesn't matter who, really), who started off the conversation by breathlessly exclaiming:

"I need to speak to someone about a network problem."

Um, who is this?

"Is this the IT department? I need to speak to the IT manager."

Um, that would be me. Who are you trying to reach? Who are you?

"I'm so-and-so, this is the number that they forwarded me to."

Um, there is no "they", we have an auto-attendant. Who is this again?

"I got your number from J. Random Otherperson."
I don't know them, but OK. Who are you?

"I'm so-and-so, and I'm with an IT recruiting firm, and I wanted to know if blah blah you had any projects blah blah" (Yeah, I figured this out by now, but I wanted to let it play out.)

Hi, well, why did you give me this whole story instead of just coming out and saying it? I don't like being told stories to. To tell you the truth, we're not inclined to want to work with people who lie to us. I certainly don't like being told a whole cock-and-bull story to get my attention. Thank you very much. Good-bye. <click>


I certainly would have listened and been polite and told the recruiter at the outset that no, we're not interested in talking to recruiters right now (and I know it's a very tough market for them, I really do not belittle their pain) but sleazy sales tactics in a field where success is defined a whole lot by trust just doesn't seem like a good plan--I suppose if you're using the "spammer" mentality of "try 100,000 and if 0.01% gets through, that's 10 sales" it might work, if you only had a short-term goal. But the good recruiters I've dealt with (and I've dealt with quite a few) spent time to cultivate a relationship of trust with clients, both on the buy and sell sides (i.e. employers and potential employees).

I hope it's not the same way in every sales arena.

November 20, 2009

Changes to 1&1 IMAP folders, etc.

Heads up 1&1 users:

If you use IMAP to access your email at 1and1.com, you'll need to subscribe to a new folder called "Sent Items" to replace your "Sent" folder, and a new "Spam" folder in addition to a "Junk" folder (if you had it).

If you use Thunderbird, you should get the FolderFlags extension and use it to set the flag on your "Sent Messages" folder to "sent"

You may also need to check to see if you're subscribed to the new folders (if it doesn't appear automatically for you) and you may also want to set the 'Junk' flag on the 'Spam' folder. You'll also want to make sure that you configure Thunderbird to save your messages automatically in the new folder under Tools | Account settings:

If you use Outlook, you have to make substantially similar changes...

Of course, the email from 1&1 doesn't mention anything (read on)...and don't get me started about the whole "2.0" nomenclature (tagging along with "Web 2.0 Expo" that I just attended...that's a subject for a Random Rant)

Continue reading "Changes to 1&1 IMAP folders, etc." »

July 22, 2009

SQL server migration lessons learned

(Boy it's been a while.)

So a few weeks back I had the pleasure of migrating a cilent's website from a shared web host + shared database host to a single VPS (virtual private server) where both the webserver and database server were on the same machine.

This consolidation is actually counter to the standard multi-tier architecture:


          +----------------------+
          |      webserver       |
          |                      |
          +----------------------+

          +----------------------+
          |       database       |
          |        server        |
          +----------------------+

In this case, we performed the consolidation because the shared database server that was available was horribly overburdened, and also the current application wasn't so much "architected" as much as "pieced together à la Frankenstein's monster" and wasn't uniformly careful with cleaning up its database connections, or even using database pooling (for .NET), so at times we'd see up to 4000 (!) separate connections attempting to be made to the database server.

While we were having awful problems with the database server, the technical staff at the hosting provider in question were helpful and responsive, even if they couldn't help us narrow down all the problems--some of which were caused, no doubt, by the client's own applications.

To tie in to the title, one of the big hangups came when we attempted to restore a MS SQL server backup of the original databases onto the local installation. One of the problems is that SQL server has a parallel notion of user IDs, namespaces (for tables and tablespaces and whatnot) and schemata (schemas), with some apparent overlap.

DISCLAIMER: I am not a SQL server database administrator (DBA) by any definition except by dint of having to manage a database on a single small server. Oracle, MySQL, PostgreSQL, these I can deal with, but SQL server? Not really.

So after restoring the databases, I found some standard permissions and ownership errors. And trying to just create a database user using the standard GUI method didn't work. (It never does. Why would anyone be surprised with this?). I found, however, that by dissecting the SQL that the create-a-user script uses, and modifying it ever so slightly, I could get things to work just my way. Perhaps this isn't the Microsoft way of doing things, and anyone who wants to disabuse me of my bad methodology should kindly step up and do so, as I am willing to learn.

The magic (line numbers inserted by me, actual databases and usernames modified, of course)

1  USE [master]
2  GO
3  CREATE LOGIN [loginname] WITH PASSWORD=N'password', DEFAULT_DATABASE=[defaultdb], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
4  GO
5  USE [defaultdb]
6  GO
7  ALTER USER [loginname] WITH  LOGIN=[loginname]
8  GO
9  USE [defaultdb]
10 GO
11 ALTER USER [loginname] WITH DEFAULT_SCHEMA=[schemaname]
12 GO
The key was that the loginname and the schemaname, even though they are the same text string, are totally independent, and needed to be assigned. But in line 3, the standard CREATE LOGIN creates a disabled user, and that broke the ALTER USER statements in lines 7 and 11. (Or at least, that's what my memory has.)

December 16, 2008

The Great Joys of MySQL permissioning

(I was going to talk about God's sick sense of humor, but that's a Random Rant for another day.) I just had some great...fun...with MySQL permissions and I thought I'd record it here for posterity, because sure as shootin' I'm going to be bitten by this problem again. I have a small script that gets called by a web page that tries to create new databases and assign permissions to yet other users from the original site...that is to say:
+-------------+       +---------------+
| web server  |       | database srvr |
| user 'user1'| ----> |               |
+-------------+       +---------------+

Now when the web page runs, it calls a script that connects as user 'root' from the webserver host.

+-------------+       +---------------+
| web server | | database srvr |
| user 'root' | ----> | |
+-------------+ +---------------+

Presumably, if user 'root' can log in, it can create and grant privileges? Ah, not so! It turns out, you can, but if you're not careful when you first set up the permissions for root@'webserver', you end up with some permissions to do things and some NOT.

The light went on when I logged in interactively from the web server and saw what I thought "remote root" could do:


mysql> show grants;
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for root@webserverhost                                                                                                                                      |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'webserver' IDENTIFIED BY PASSWORD '*you think i will put this here??!!' WITH GRANT OPTION                                   |
| GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON `mysql`.* TO 'root'@'webserverhost'          |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+

When the light went on, it nearly blinded me.
Notice that in the second line, the "remote root"'s privileges on the mysql database (where all of the user privileges are kept) has no 'IDENTIFIED BY' (meaning, a password is required, and has a cryptographic has value equivalent to what I've deleted).
In fact, you won't ever see the 'IDENTIFIED BY' in the second line, but what tickled me about this is that there's no indication that the "remote root" user wouldn't be able to grant permissions.
There's a paradox there. In order to change privileges on the database server (to allow 'user1' to log in, for example), you need to provide no password (that is, you must not provide a password), but in order to log in, you must provide a password! (It is only because I've seen this behavior before that I recognized this; there is otherwise little other indication about it.)
MySQL, to their credit, does document most of this, in their typical fashion, but without any mention of what workarounds might be necessary, or that the regular 'GRANT' facility might not work the way you think, or where the command will succeed--it will do what you tell it to--but not what you want it to... Once I rectified this by granting privileges on the mysql table to the "remote root" user, all the problems went away:
mysql> grant all on *.* to root@'webserver' identified by 'xxxxxyyyy' with grant option
    -> ;
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

and then...
[jbaltz@webhost] >mysql -u clover -pxxxxxyyyy -e 'show tables from newDataBase' -hDatabaseServer
+-----------------------+
| Tables_in_newDataBase |
+-----------------------+
| User                  |
+-----------------------+

...which gives me what I need.
Once again: MySQL acts like you tell it to, but not how you might want it to, and there's no indication of how you might shoot yourself in the foot here; it simply silently sets things up in an impossible fashion.

September 26, 2008

Trying to hire...again...

In this economy? Sure.

I'm doing craigslist again (see here or here) but I'm also trying out now the Joel on Software jobs board (see our ad here)

For what it's worth, I posted the CL ad yesterday and it took over an hour for the first person who didn't read the ad to spam me.

I have to admit to reading the CL gigs sections occasionally to try to find new clients, but I don't wildly spam every posting on the board. It's a little insane.

September 4, 2008

First few impressions of Google Chrome

Following the maddening crowds, I downloaded Google Chrome and installed it on my Windows machine. My first impressions are:
  1. Javascript-heavy sites like Facebook, Jango, LinkedIn and Gmail itself feel much faster, whether they actually are is somehow irrelevant...I've not done any real testing of pageloading or rendering speed versus Firefox 3 (my default browser); my observations are totally subjective.
  2. Ack! Where did all these advertisements come from? After having been spoiled by Adblock in Firefox, browsing sites in Chrome seems to be so ... much noisier.
  3. The menu-barless top is something to get used to; Mac users have had it forever (MacOS has always had the menu bar pinned to the top of the screen), but for X-windows and Windows users, it's a new and somewhat disorienting feeling.
  4. The "incognito" window is an interesting feature-let that should definitely be stolen by Firefox, while it doesn't prevent being spied upon by corporate firewalls or proxy servers, it does at least provide a way to prevent having things saved on your system that you might not want to find later. (Of course, no system is perfect, but at least this provides the thinnest veneer of deniability...)
  5. Otherwise the look-and-feel isn't significantly different than the tabbed browsing in FF3, with the exception that there's no longer a "home page" (or set of pages) any more, just a thumbnail of your most recented sites.
I'm not a GoogleDocs user, so I'm not really the target audience for Chrome, but it's got that new-car smell and go-faster stripes that make the JS-heavy sites that are so prevalent and growing in complexity and pervasiveness. (There are still quite a number of non-JS-heavy plain-HTML sites out there--this blog being one of them--so there's some time for every other browser to keep up.)

August 18, 2008

Verizon FIOS is (no longer) one hop away from the whole world

Well, a little while ago I noticed that Verizon was breaking traceroute (a very useful network debugging tool).

Well, now, Verizon seems to have seen the error of their ways (?!) and allowed us to see our network paths:

-bash-3.2$ /usr/sbin/traceroute www.google.com
traceroute: Warning: www.google.com has multiple addresses; using 64.233.169.103
traceroute to www.google.com (64.233.169.103), 30 hops max, 40 byte packets
 1  fw-gw.3phasecomputing.com (192.168.xxx.yyy)  0.673 ms  0.532 ms  0.511 ms
 2  98.113.45.1 (98.113.zzz.aaa)  5.012 ms  4.208 ms  4.495 ms
 3  G4-0-0-1955.LCR-09.NYCMNY.verizon-gni.net (130.81.137.34)  5.021 ms  5.199 ms  5.033 ms
 4  130.81.29.236 (130.81.29.236)  5.297 ms  5.569 ms  5.028 ms
 5  0.so-4-3-0.XL4.NYC4.ALTER.NET (152.63.10.29)  5.561 ms  5.464 ms  5.837 ms
 6  0.ge-5-1-0.BR3.NYC4.ALTER.NET (152.63.3.118)  7.157 ms  6.812 ms  6.638 ms
 7  te-10-2-0.edge2.NewYork2.level3.net (4.68.110.233)  14.080 ms  14.803 ms  13.822 ms
 8  vlan69.csw1.NewYork1.Level3.net (4.68.16.62)  19.441 ms vlan79.csw2.NewYork1.Level3.net (4.68.16.126)  15.586 ms vlan89.csw3.NewYork1.Level3.net (4.68.16.190)  24.895 ms
 9  ae-74-74.ebr4.NewYork1.Level3.net (4.69.134.117)  23.574 ms ae-84-84.ebr4.NewYork1.Level3.net (4.69.134.121)  17.200 ms ae-74-74.ebr4.NewYork1.Level3.net (4.69.134.117)  16.937 ms
10  ae-3.ebr4.Washington1.Level3.net (4.69.132.93)  24.887 ms  17.200 ms  18.345 ms
11  ae-94-94.csw4.Washington1.Level3.net (4.69.134.190)  20.201 ms ae-63-63.csw1.Washington1.Level3.net (4.69.134.162)  15.321 ms  14.534 ms
12  ae-1-69.edge1.Washington1.Level3.net (4.68.17.16)  134.966 ms  13.450 ms  13.546 ms
13  GOOGLE-INC.edge1.Washington1.Level3.net (4.79.231.6)  13.812 ms GOOGLE-INC.edge1.Washington1.Level3.net (4.79.228.38)  13.720 ms GOOGLE-INC.edge1.Washington1.Level3.net (4.79.231.6)  14.268 ms
14  64.233.175.171 (64.233.175.171)  14.524 ms 64.233.175.169 (64.233.175.169)  14.088 ms  14.066 ms
15  216.239.49.149 (216.239.49.149)  16.987 ms 216.239.49.145 (216.239.49.145)  17.781 ms 216.239.49.149 (216.239.49.149)  17.519 ms
16  yo-in-f103.google.com (64.233.169.103)  14.319 ms  13.705 ms  14.092 ms

August 13, 2008

MSN "spim" growing.

So...have you seen the old new MSN spim (Spam for IM) going around? It looks like:
(2008-08-13 09:32:10) albaketapy@hotmail.com: Hey Jerry%20B.%20Altzman .....I cant upload my pics to msn for some reason! Hit me back up on http://xxxxxx.blogspot.com

(I've obfuscated the first part of the blogspot URL, since I don't want to drive traffic there.) Evidently, I'm not the only one who's seen this, and it's not particularly new, but no one seems to have a good solid idea what this is. I've received about a dozen of these, all from Hotmail (MSN) addresses:
$ grep -cri 'Hit me back up' *|grep -v '0$'
agnessopyby@hotmail.com/2008-08-11.032149-0400EDT.txt:1
albaketapy@hotmail.com/2008-08-13.093210-0400EDT.txt:1
annefogabem@hotmail.com/2008-08-11.103216-0400EDT.txt:1
elisecokaw@hotmail.com/2008-08-12.182304-0400EDT.txt:1
genevievenugimox@hotmail.com/2008-08-12.231241-0400EDT.txt:1
jennylevyv@hotmail.com/2008-08-13.032007-0400EDT.txt:1
lessielydoc@hotmail.com/2008-08-10.235141-0400EDT.txt:1
lorenanunecaz@hotmail.com/2008-08-12.204747-0400EDT.txt:1
nanettepusun@hotmail.com/2008-08-11.080932-0400EDT.txt:1
nanettepusun@hotmail.com/2008-08-13.070848-0400EDT.txt:1
phoebecytol@hotmail.com/2008-08-11.054531-0400EDT.txt:1
robertcopow@hotmail.com/2008-08-12.155737-0400EDT.txt:1

July 21, 2008

Verizon FIOS is one hop away from the whole world

Wow. I knew my Verizon FIOS was good, but not this good. Evidently, it’s exactly one hop away from every destination I should like to traceroute to...
$ tracert -d www.jbaltz.com

Tracing route to www.jbaltz.com [74.208.29.13]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.xxx.yyy.zzz
  2     6 ms     4 ms     4 ms  98.113.aaa.bbb
  3    45 ms    45 ms    45 ms  74.208.29.13

Trace complete.
Hrm...1&1 is one hop from my firewall? Rockin’!
How far am I from some random service provider?
$ tracert -d mail.emailsrvr.com

Tracing route to mail.emailsrvr.com [207.97.245.100]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.xxx,yyy.zzz
  2     6 ms     5 ms     4 ms  98.113.aaa.bbb
  3    14 ms    14 ms    14 ms  207.97.245.100

Trace complete.
Yow! Of course, this is Windows traceroute. From a FreeBSD box, I get somewhat different results:
[jbaltz@iridium ~]$ traceroute -n www.jbaltz.com
traceroute to www.jbaltz.com (74.208.29.13), 64 hops max, 40 byte packets
 1  192.168.xxx.yyy  0.514 ms  0.359 ms  0.338 ms
 2  98.113.aaa.bbb 4.572 ms  5.229 ms  4.341 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
(18 more lines like this deleted...)
Clearly I’m not the only one seeing this problem, although I am not sure that the link I posted is really relevant...

February 21, 2008

Clown (er, hiring) update

Update on last night’s posting:
As of this morning, I’ve received:
  • One person who followed instructions and sent in a (good looking) CV in the proper format.
  • One clown from netzero.net who sent in an MS word document
  • One clown from India responding as an outsourced PHP programmer—I did say “on-site”, did I not?
  • One clown from India hawking website and graphic design services.
  • One clown offering his outsourced-to-India “audited by KPMG to be CMMi Level 5” solution
  • One clown who writes, twice, in part:
    I thank you for the opportunity given to quote for the above and take pleasures in forwarding our resume in simple.

    Why am I ready for this project?

    I am a Service Exporter. I must export service.I do Export My ability.I do it really happily. I am enjoying working with php. I am a Lecturer for php in local computer institute.

Note that I do not poke fun of people for their English writing (in general, although the snippet above is quite amusing), but I do fault people for spamming and blast-responding to every post in Craigslist, and, of course, for not reading. It is one of the things that strongly reduces the value of a posting there—although for $50 for a month, it’s not bad. I suppose I should try Joel Spolsky’s site now.

February 20, 2008

More job postings...more clowns

So I made another posting for a job opening at 3 Phase tonight (link will only be good for 30 days), this time for a part-time PHP programmer.

The posting went up at 2312 EST tonight. The posting said “send résumé in HTML or plain text” and it also said “must be able to read and follow directions”.

Email at 2355 LCL came in with a Word document attached.

43 minutes from FIRST POST to first clown. I beat my previous record.

Sigh. C’est la guerre.

January 25, 2008

PIX IPsec VPN problems

So you have a PIX and you want to set up an IPSec LAN-to-LAN VPN with it and you're having major troubles.

Specifically, you're seeing:

%PIX-3-713119: Group = xxx.yyy.aaa.zzz, IP = xxx.yyy.aaa.zzz, PHASE 1 COMPLETED
%PIX-3-713902: QM FSM error (P2 struct &0x1c0bd30, mess id 0x4a08f6c8)!
%PIX-3-713902: Group = xxx.yyy.aaa.zzz, IP = xxx.yyy.aaa.zzz, Removing peer from correlator table failed, no match!
%PIX-4-113019: Group = xxx.yyy.aaa.zzz, Username = xxx.yyy.aaa.zzz, IP = xxx.yyy.aaa.zzz, Session disconnected. Session Type: IPSecLAN2LAN, Duration: 0h:00m:00s, Bytes xmt: 0, Bytes rcv: 0, Reason: Phase 2 Mismatch
A few things you need to know about the PIX and its IPSec VPN implementation.
  1. The PIX is very picky about every parameter matching.
  2. The PIX has PFS turned off by default for L2L VPNs
This means that if you’re running something like the OpenBSD IPSec VPN (with isakmpd) which is otherwise pretty permissive about parameters, you must explicitly turn off PFS by turning off DH group 2 in the ipsec.conf file:
  • OLD:
     quick auth hmac-md5 enc 3des group modp1024\
  • NEW:
     quick auth hmac-md5 enc 3des group none \

January 6, 2008

Subversion for site control, take two

Rather than write about the NYT magazine article on electronic voting machines and their failures that everyone else is writing about (or simply pointing to) I want to share a bit of an email exchange I had with someone who posted to CraigsList for “subversion help”. (No, this is not meant to be something the Sedition Act would get me on, but rather the version control system of that name.

Just to get this out front, since some people who know me might think this is meant to be mocking of my correspondent: it most certainly is not in any way meant to be disparaging to anyone, only enlightening (I hope).


He starts with:

How do we protect our trunk from bad coders/commits without spending too much time in administration, merging, Etc. Most of the developers will be new to SVN. There will be at least a dozen developers, several with language and communication barriers. Our site is live and needs lots of bug fixes so quick response is needed as well as protection of the trunk against bad code.

This isn't really a 'subversion' question, strictly speaking, as it really is more of a 'process' question. You'd encounter much the same problems in ANY version control system you used.

Continue reading "Subversion for site control, take two" »

November 27, 2007

Secrets of the Intel SS4400-E (aka SS4000-E)

We recently purchased an Intel SS4400-E network-sttached storage (NAS) unit
yes, that link points to an SS4000, but that seems to be the only way the product is listed; ours identifies itself as a 4400 and this as caused me no end of trouble in discovering any information about the unit) and put some 500GB SATA drives in it to create a nice terabyte-sized RAID-10 array.
One of the nice things about this particular device is that it supports USB drives as well; you can hang a USB-drive (like the Western Digital MyBook) and the unit will (allegedly) serve it up either via NFS or CIFS shares. The idea is now I can take this big ol' USB drive, and instead of attaching it directly to a machine, I can make it “network attached” and can use it to back up the various and sundry Windows machines in the office.


Well, except when it doesn’t.


The disk had been previously formatted as UFS2 for a FreeBSD server we have here in the office, but we discovered an interesting problem with FreeBSD 6.2 and USB devices and our particular server.

It turns out that for our server, and FBSD 6.2, the machine would not reboot cleanly with the USB drive attached — it would hang at the BTX loader. This might be a known problem, maybe (?) related to ACPI, and is hopefully going to be fixed in 7.0


Well, the NAS unit would NOT recognize the drive when it was plugged in. At all. Ever. So, I figured “maybe if I formatted it as NTFS, which is much more common, the machine would like it more better.”

It did not.


Then, I found this forum link here, that told me about a mysterious URL I could go to on the server to turn on an ssh server — the box runs a special Linux distribution under the hood. You simply browse to http://hostname/ssh_controlF.cgi, and voilà, you’ve got the ability to ssh into the machine:


[jbaltz@aaa ~]$ ssh kr -l root
root@xxx.yyy.zzz's password:
Welcome to

_/_/_/ _/_/_/ _/_/_/ _/
_/ _/ _/ _/ _/_/_/_/ _/_/ _/ _/_/
_/ _/_/_/ _/_/ _/ _/ _/ _/_/
_/ _/ _/ _/ _/ _/ _/
_/_/_/ _/ _/_/_/ _/_/ _/_/ _/

Powered by FalconStor Software, Inc.
Copyright 2001-2006 by FalconStor.
All Rights Reserved.
http://www.falconstor.com
#



WARNING


If you follow any instructions of mine beyond this point, or even if you don’t follow them, you might find yourself having destroyed your array, lost your data, your car might break down, your wife might leave, your dog might die, etc. etc. etc. Read all you want, but if you type in commands any of these commands, you do so at your own risk!
You have been warned.



This is good news! Linux, I understand. After logging on, I see what the machine thinks it should be doing when I plug in the USB drive:

VFS: Can't find ext3 filesystem on dev sdf1.
VFS: Can't find an ext2 filesystem on dev sdf1.
XFS: unknown mount option [gid].
FAT: bogus number of reserved sectors
VFS: Can't find a valid FAT filesystem on dev sdf1.
usb-storage: device scan complete

Hooray! I see what the problem is (although when I realized the machine ran Linux, the light went on right away): the machine only understands the old FAT-32 (native mounts of NTFS partitions on Linux is just now getting read/write support) and native Linux filesystems, like ext2, ext3 and xfs.

So....I put the drive back on the machine, and a view in the /var/log/messages file showed me that the machine thought the USB disk was found on /dev/sdf


XFS: unknown mount option [gid].
FAT: invalid media value (0xf3)
VFS: Can't find a valid FAT filesystem on dev sdf.

and all I needed to do now was create a new partition

# fdisk /dev/sdf

The number of cylinders for this disk is set to 60801.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdf: 255 heads, 63 sectors, 60801 cylinders
Units = cylinders of 16065 * 512 bytes

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-60801, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-60801, default 60801):
Using default value 60801

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: If you have created or modified any DOS 6.x
partitions, please see the fdisk manual page for additional
information.
Syncing disks.


and then create a partition. I tried making an XFS partition first:

# mkfs.xfs -f /dev/sdf1
meta-data=/dev/sdf1 isize=256 agcount=16, agsize=7631000 blks
= sectsz=512
data = bsize=4096 blocks=122096000, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks
realtime =none extsz=65536 blocks=0, rtextents=0

and mounted it on the known path for USB disks:

# mount /dev/sdf1 /nas/usbdisk1
# df -h
Filesystem Size Used Available Use% Mounted on
/dev/md0 248.7M 95.2M 140.6M 40% /
/dev/vbdi2 195.3M 152.0k 195.2M 0% /nas/NASDisk-00002
/dev/vbdi3 195.3M 260.0k 195.1M 0% /nas/NASDisk-00003
/dev/vbdi4 930.2G 183.6G 746.7G 20% /nas/NASDisk-00004
/dev/sdf1 465.6G 272.0k 465.6G 0% /nas/usbdisk1

Hey hey hey! Some small success here! (It is sobering to think that the disk space I have available here that I purchased for about USD 1,000 cost Omnipod about USD 2,000,000 in 2000.)
Can I access this now from my windows machine, which was the original idea? Well, kinda sorta. Not right away. I now go right to the smb.conf file, and edit the section on the usbdisk

[usbdisk1]
comment =
path = /nas/usbdisk1
max connections = 0
read only = no
browseable = yes
comment =
valid users = root,jbaltz,guest,
write list = jbaltz,
available = YES
nt acl support = no

(At this point, I’ve probably gone beyond the point of no return on the unit, but I need this to work.) I notice that the mount point itself needs to be writable by the unix group my windows user maps to:

# ls -dl /nas/usbdisk1
drwxr-xr-x 2 root root 8 Nov 27 22:46 /nas/usbdisk1
# chgrp nasgrp /nas/usbdisk1
# chmod g+w /nas/usbdisk1

Can I use it? Yes I can!

jbaltz@cesium /home/jbaltz
$ net use z: \\\\krypton\\usbdisk1
The command completed successfully.

(Three cheers for the cygwin bash shell.) And off I go:

jbaltz@cesium /home/jbaltz
$ cd /cygdrive/z

jbaltz@cesium /cygdrive/z
$ df -h .
Filesystem Size Used Avail Use% Mounted on
z: 466G 272K 466G 1% /cygdrive/z

jbaltz@cesium /cygdrive/z
$ mkdir foo

jbaltz@cesium /cygdrive/z
$ cd foo

jbaltz@cesium /cygdrive/z/foo
$ cat > bar
hi there

jbaltz@cesium /cygdrive/z/foo
$ ls -l
total 1.0M
-rw-r--r-- 1 jbaltz None 9 Nov 27 23:20 bar

Note that my one little 9 byte file appears to take up 1 megabyte on disk!

However—mirabile dictu—I appear to be done! I still have a daunting task ahead of me: tightening up security somewhat. Yet, this problem seems to be now behind me.

For posterity, I should note that there is a wealth of interesting information and utilities in the /usr/local/ipstor directory tree.

September 20, 2007

SSH from within a FreeBSD jail gotchas

If you’re using FreeBSD’s jail(8) mechanism (which, by the way, is similar to Solaris zones but not as fancy or as featureful), you might have occasionally seen this problem:

You’re inside the jail, and you’d like to ssh out, and you get:

[jbaltz@boron ~]$ sudo jexec -u jbaltz 1 bash
[jbaltz@xxx /]$ ssh localhost
socket: Protocol not supported
Host key verification failed.
[jbaltz@xxx /]$

...and you make ssh more verbose, and you see the following towards the end:

[root@xxx /]# ssh -v localhost
OpenSSH_4.5p1 FreeBSD-20061110, OpenSSL 0.9.7e-p1 25 Oct 2004
debug1: Reading configuration data /etc/ssh/ssh_config
...
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: read_passphrase: can't open /dev/tty: Device busy
Host key verification failed.

...the problem is probably that you’ve jexec(8)’d into the jail, instead of logging in. SSH expects to be able to attach to a tty, and when you connect in via jexec, you don’t create one:

[jbaltz@boron ~]$ sudo jexec 1 bash
[root@xxx /]# who
[root@xxx /]#

wupsie! The solution is to log into the jail “the regular way” via ssh:

[jbaltz@boron ~]$ ssh xxx
Password:
Last login: Thu Sep 20 14:54:04 2007 from xxx.3phasec
[jbaltz@xxx ~]$ ssh localhost
socket: Protocol not supported
The authenticity of host 'localhost (127.0.0.1)' can't be established.
DSA key fingerprint is b6:d7:47:4b:25:60:75:36:2e:30:22:2f:27:ba:67:27.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (DSA) to the list of known hosts.
Password:
Last login: Thu Sep 20 14:54:31 2007 from xxx.3phasec
[jbaltz@xxx ~]$

and voilà, it works!

September 11, 2007

Infosecurity NY expo visit

So Catherine (partner-in-geek here at 3 Phase) and I went today to the Infosecurity New York trade show, which this year was co-located with ISC East. ISC is the International Security Conference & Expo: it's everything you wanted to see from electrical wiring to electronic door locks to barriers that pop up out of the ground to keep you from driving through to wireless speakers to (it seems) dozens of CCTV-over-IP solutions. It was all quite interesting, and it made sense to put infosecurity and physical security next to each other. All too often those of us in the computer field who worry about virtual security forget about the simple things we need to do to secure the data: keep it in a locked cabinet, behind a locked door, with limited access to the general public.

Another side-effect of this happy conflation is that the IT side tends to avoid the hardware issues that actually do affect clients, like power outages, so I actually got to see the very hard hardware side (finding the UPS and, just as important, battery salesman and suppliers) However, the last time I went to Infosecurity NY, there were a number of other networking equipment (read: firewall) vendors there, and this year barely any. (No Juniper/Netscreen, no Cisco/PIX/ASA, no Foundry, just Fortigate from what I could find.)

A few years back—undoubtedly a sign that I’m getting older—I decided that I would go to trade shows to actually see what people are selling and seeing what the near-state-of-the-art is, instead of just collecting swag. (Of course, there was some nice swag to be had, but I missed out on it.) By and large, the exhibitors were interested in selling appliance solutions for security folks like Barracuda Networks and StopSpamNow.com (I cannot remember offhand exactly who they were and don’t care to put a link in the blog for them.) were two of just the anti-spam plugins. It seemed like there were well over half a dozen IDS vendors selling plug-in IDS (Intrusion Detection Systems) solutions, and this doesn't count the firewall vendors (Fortinet, e.g.) who provide integrated IDS into their firewall unit.

There were a few, proud, software solutions vendors—one that impressed me some was SafeBoot, who gives you a pre-boot authentication environment to decrypt the contents of a hard-drive (I imagine your anti-virus software vendor must have a good time with that!)

It was also gratifying to be able to speak occasionally to some of the engineers of products I currently use to find out that the features that were broken 18 months ago finally got fixed. Plus, if you push the salespeople enough, you can actually get them to do the unspeakable: compare themselves honestly against their competition. Things like:


Oh, you want to do <XXX>? Hrm, well, that’s not a feature we really specialized in...if you want that, you might want to talk to vendor <YYY> you’ll be more satisfied overall

This, of course, only works when you can speak to a techie. Most of the time they’re hidden away, brought out only when you can prove your street creds and stump the salesperson.

Of course, being a trade show, some things still are bothersome:


  • You actively have to avoid the gaze of salespeople, lest they come down upon you and pounce and scan your badge and subject you to years of junk-real-mail. This happened to me as soon as I walked onto the trade-show floor; someone basically walked up to me, scanned my badge, and told me “You don’t want that bag, those handles break. Take one of mine.” (His bag is nicer, but he still crossed the line of civility, in my opinion.
  • Some salespeople just seem to not understand subtle cues that I am not interested; they’re inured to them, I suppose. Instead, I have to resort to more traditional techniques: “I don’t think this product is right for my clients, thanks. Good bye. No you may not scan my badge.”
  • Some vendors send no one technical at all; just some marketing folk and new-and-clueless ones at that. It’s even more jarring when it’s a vendor of a product that I use and recommend.

May 31, 2007

Trixbox and call waiting

So I have one client who just migrated from using Trixbox (neé Asterisk@home) in a VMWare as his telephony platform (along with a bunch of Grandstream SIP phones) the newest version (2.2) natively on the machine. (They were experiencing the things you'd expect by running a timing-based application from within VMWare—bad sound quality, odd playback of audio files, etc.)

So we backed up the old data stores, installed Trixbox natively, and things went about as smoothly as you could expect.

One function that had never worked out of the box, for some reason, was call waiting: even though users would enter *70 to enable call waiting, for some reason the asterisk DB (an oldy moldy Berkeley DB database) wasn't taking the change.

The solution was to run the following from the asterisk console (asterisk -vr):

For each extension, execute the following command

database put CW <extension> ENABLED

...and that does the trick!

March 4, 2007

MySQL resiliency after a crash and relay log files

One of my clients had a major power failure at his (new!) data center the other week, and one of the side effects was that there was quite a bit of data corruption (on an ext3 filesystem). One thing we lost was the LDAP database (that we recovered from the LDAP slave) and one file that got lost was a MySQL relay log file for a MySQL "slave".

MySQL appears to use these "relay log" files as a buffer between reading from a master server and feeding into a slave—it's supposed to make things more resilient.

Of course, if you end up with a missing file, you end up with the following in your mysqld.log file (on Linux):

070304 22:24:43 [ERROR] Failed to open the relay log './<host>-relay-bin.000057' (relay_log_pos 494316200)
070304 22:24:43 [ERROR] Could not find target log during relay log initialization

This error will prevent you from starting up a slave, and will give you an error:

ERROR 1201 (HY000): Could not initialize master info structure; more error messages can be found in the MySQL error log

The solution that I found that finally worked was to:


  1. Create a new dump of the master database, FLUSH TABLES WITH READ LOCK, recording the master log position, etc.
  2. Load in the dump into the slave. (Thus far we're following regular directions for creating a slave.)
  3. Stop MySQL.
  4. cd /var/lib/mysql; rm *relay-bin* master.info (This gets rid of all the old information about the old slave info.) (DANGER: Like all other instructions to delete files, use this at your own risk. Make backups of everything if you're unsure.)
  5. Restart MySQL
  6. Execute the CHANGE MASTER command that you use to set up the slave

February 22, 2007

useful bash MUTEX code

Problem: you have a script that runs periodically (like from cron) and sometimes the time to complete is longer than the time in between invocations. You end up with multiple scripts running one on top of the other, and if the scripts are at all CPU-intensive, you can bring the machine grinding to a halt.

Solution: use a MUTEX (mutual exclusion) to prevent scripts from starting if one is active. Also use a timer in case one script terminates and doesn't clean up its Ps and Vs.

bash code:


  1. # add in the mutex code
  2. # we need to figure out how to NOT run too frequently, so we don't stomp
  3. # on each other.
  4. # (c) Jerry B. Altzman/3 Phase Computing.
  5. # Permission is granted to use this code but no warranty is implied whatsoever
  6. # requires gnu date
  7. MUTEX=/tmp/MUTEX`basename ${0}`
  8. NOW=`date '+%s'` # seconds since the epoch

  9. # if the mutex exists, see how old it is. If it's too old,
  10. # delete it, and start again.
  11. if [ -f $MUTEX ]; then
  12.   THEN=`cat $MUTEX`
  13.   DELTA=$((NOW-THEN))
  14.   if [ $DELTA -gt 3600 ]; then # too long
  15.     rm $MUTEX
  16.     pkill -9 $0 # will probably kill this one too
  17.   else
  18.     exit 0 # MUTEX is there and relatively new
  19.   fi
  20. else # MUTEX file is not there
  21.   date '+%s' > $MUTEX
  22. fi

  23. trap "rm $MUTEX" 0 1 2 15

The comments are pretty self-explanatory, but just in case, the idea of the code is thus:
Line 7: Define a file that will be the MUTEX for this program. It is unique to this program but shared across all invocations.
Lines 12-14: Check to see if the MUTEX exists. If it does, examine its contents—the contents are the time of the last invocation. If the time is “not too long ago” (I have it set here as one hour, but you can change that depending on your preferences.)
Lines 15-17: If the MUTEX exists, and is “old”, then delete the MUTEX file, and kill off any processes with our name to clean up any “hung” processes (and the next periodic run will start afresh).
Lines 18-20: If we are here, the MUTEX isn’t that old, so we just exit and wait for the next invocation.
Lines 21-23: Otherwise, there is no MUTEX here—either the old process exited normally or we cleared out a hung processes. In any case, create a new MUTEX with the current time.
Line 25: This “trap” statement is a bash directed that says “if this script terminates with exit codes 0, 1, 2, or 15, then perform the operation listed” which, in this case, is to clean up the MUTEX itself.

January 15, 2007

FORTRAN then Fortran and now this?

So it leaks out that Sun Microsystems is working on a Fortran replacement. It is supposed to take advantage of all the new multi-core chips being produced, moreso than the parallel extensions to F90/F95.

I don't like the name much, though. I think Sun should just take over the Fortran standards working group and call it Fortran 2010. (It will look pretty different from the Fortran 90 that I wrote my Ph.D. thesis work in; and sadly I can't even get Macsyma any more.)

In the grand tradition of saying “I don’t know what language they’ll be doing <X> in in 30 years, but it’ll be called <old name>”:

I don’t know what language LINPACK will be written in in 2025, but it will be called Fortran.

December 25, 2006

Registrars, DNS, and vanishing off the internet

So last week at this time I had a hard, nasty thing happen to a client of mine: due to some classic incompetence at Network Solutions, they vanished off the internet for about 20 hours. In order to understand exactly what happened, I need to delve a little bit into how domain name registration and the DNS (domain name system) works.

In this day and age, when you want to register a domain name (say, www.jbaltz.com), there is actually a two step process that goes on:

  1. You register a domain name with a registrar, like GoDaddy or 1and1 or Network Solutions (10 years ago NetSol was the only game in town, but that is another story.) and they verify that no one else has that domain name, and they reserve it for you.

  2. At the same time, they notify the TLD name server for your TLD with a list of the authoritative name servers for your newly-formed domain.


What? Wait? Come again? What’s all that? Let’s define a few terms:

  • A registrar is just the organization that registers your name and enforces global uniqueness—there can be no other “jbaltz.com” sites out there but this one. It may also hold “whois” information about the name of the responsible person or company are behind a domain, but nowadays many registrars will allow you to obscure your whois information to prevent onslaughts of UCE (spam).
  • A TLD (Top Level Domain) is the last part of your domain name: typically “.com” or “.org” or such, or even a country-specific domain like “.uk” (British sites like www.amazon.co.uk) or “.il” (Israeli sites, like www.huji.ac.il, the Hebrew University in Jerusalem).
  • An authoritative name server is a site that agrees to answer questions of the type: “Where do I find the IP (numerical) address of site www.example.com?” and “Who receives mail for www.whoever.com?” (It is at this point that discussions usually go into things like “SOA” and “glue records” and most peoples’ eyes glaze over, but this is actually an important contribution to the discussion.)
  • The TLD name servers are a group of systems that hold all the names in a particular TLD, and a list of who the authoritative name servers are.

To wit, for jbaltz.com, the records that the .com TLD name servers hold is:

jbaltz.com. 172800 IN NS ns27.1and1.com. 
jbaltz.com. 172800 IN NS ns28.1and1.com. 

which means that the internet hosts “ns27.1and1.com” and “ns28.1and1.com” will be able to answer the “who” and “where” questions about jbaltz.com. (The other numbers and codes are somewhat irrelevant to this discussion, although they are important.)

(Digression: A long time ago, there was actually semantic difference between “.com”, “.org” and “.net”, but nowadays the difference appears to be entirely nominal: people just scoop up the “.org” name or the “.net” name if the “.com” name is taken. There are a few TLDs that do maintain an entry-barrier other than money: “.edu” requires that you actually prove to them that you’re an educational institution, and I believe “.museum” has a similar requirement. Also, I believe other country-wide TLDs require proof of residency or something to register a website with them, with notable exceptions being Tuvalu “.tv” and Western Samoa “.ws” )

If you’re a typical website hosting with your provider (like 1and1, which is the hosting provider for this site), your hosting provider may act as your registrar (holding your name in the global namespace of .com and telling the TLD nameservers who is the nameserver for your domain) and act as the authoritative name server for the domain, but they do not have to do so. jbaltz.com is registered through MelbourneIT (neé www.registerfree.com) but has its domain name service provided through 1and1. Many many other sites do that.

My client’s site was one of them.

He had registered his site through Network Solutions, but another site (his hosting provider) was the authoritative DNS for his domain. He was moving from one hosting provider to another, and in the interim it made sense to make Network Solutions his authoritative DNS, right? I mean, they already have his registration, and they have an easy web-based interface to set up the DNS entries that were needed. It seemed like the easiest way to have a smooth transition from one place to another.

Now, Network Solutions, oddly enough, does not make moving back to them for name service easy. You cannot set up all your various and sundry domain names (www.this.com, www2.this.com, mail directions) beforehand and then tell them “OK, we want NetSol to be the authoritative DNS for us, in addition to being our registrar.” Instead, you have to do it in two steps:

  1. Move your DNS back to NetSol
  2. Set up your DNS and all its addresses in high-speed.

Going on behind the scenes several things are going on: NetSol is setting up its own servers to be equipped to answer questions about the new domain, and NetSol is informing the TLD nameservers that it is going to be authoritative for the new domain. The former process is generally pretty quick, and the latter process can be time-consuming. (You are typically told that it takes 24-48 hours, although in reality 6 hours is about how fast it works for .com.)

What has happened now? We moved the DNS back and NetSol did the following: it notified the TLD nameservers that it was now authoritative, but it did not actually configure its own name servers to answer questions!

I think you can see where this is headed.

Now, after the move, it turns out the TLD nameservers were updated, mirabile dictu, almost immediately. NetSol’s own nameservers, however, were not updated. Which means the following things happened:

  • A user out on The Vast Internet tried to find “www.jerrysclient.com

  • The user’s ISP’s nameserver asked the global nameserver who was responsible for www.jerrysclient.com. The global TLD nameserver replied: “NetSol is”

;; ANSWER SECTION:
jerrysclient.com.  3699    IN      NS      NS15.WORLDNIC.com.
jerrysclient.com.  3699    IN      NS      NS16.WORLDNIC.com.
  • NetSol, of course, denied knowing anything about this domain, and said, in return, “go ask the root”.
  • The root said “go ask NetSol”, and we get a nice little infinite loop.
  • Eventually, the name query would time out, and no one could find my client’s site, and poof they have vanished off the internet!

Calling up Network Solutions technical support (“For a painful experience, press 1. To be on interminable wait, press 2”—I’m sure that Scott Adams had this in mind when coming up with Dogbert’s tech support.) was less than useful: they tried at great length to convince me that I simply had to wait for this information to propagate through the internet. I replied that it, indeed, had propagated, and the ball was now in Network Solutions’s court, and could I pretty please speak to someone in their DNS services group (I thought about posting something inquisitive to NANOG but decided later that it would be more efficacious to just wait.) and of course, I was told, I could not, but that he could enter a ticket for me, and the problem, being NetSol’s, should “clear up in 2-3 hours, tops”. The president of the client firm spent several fruitless hours, getting escalated up a never-ending chain of bureaucrats until he finally just got fed up. After about 20 hours, NetSol finally got their act together, and the site finally came “back to Earth”.

And there was much rejoicing.

December 1, 2006

MS Office 1, OpenOffice 0

So last night (earlier this morning) I need to write an envelope. My handwriting is atrocious at 1 p.m., and it’s 12 hours worse at 1 a.m., so I turn to my word processor to do it for me.

On my current, new, laptop, I don’t have MS Office installed, but I do have OpenOffice 2, so I fire that up to see what it can do.

I find my way down to the envelopes composer, type in the addresses, and then try as I might, I cannot find the exact envelope orientation I need for my printer. I find one that, by all rights, should work, put an envelope in the feeder, and click print.

What happens next is that the envelope feeds through, then a plain piece of paper gets the envelope text—evidently, the envelopes are printed only after the main text (of which there is none). Two or three iterations of trying to get the right order (remember, it is 1 a.m.!) and I throw up my hands in despair. Why can’t it just take the envelope first?

I go to the other computer in the office, with MS Word 2007 installed, and go to the Envelopes wizard. It looks like the one I’ve used countless times since I started using it in Office 2000, enter my addresses, select the correct envelope orientation, insert the envelope in, and go. Time start to finish is about 2 minutes, including changing the default fonts for the envelopes. (I’m no fan of Arial to be honest.)

It isn’t that OpenOffice does it so wrong, or can’t be convinced to do it right, it’s that Microsoft made it easy and straightforward to do it right—if I want the envelope to be attached to the document, I can have that, or I can just print it out by itself. (There’s a big “print” button there on the envelope wizard.)

Some things, believe it or not, Microsoft does right.

November 3, 2006

Need volunteers for a site I'm working on...

So one of my clients is building a dating website, and we need some playtesting of the profile gathering process. This is also so we can build up a corpus of profiles upon which we can ply our special sauce matching method.

So if you've got 10 minutes, browse to this page (yes, I know it's just to an IP address; don’t worry, it doesn’t take any real information that could be used against you), and provide some feedback as to the method of data gathering. (What did you think of the questions? Of the choices?)

Note that the user interface is not going to end up exactly as ugly as it is now. The interest is mostly about the questions, types, and order.

October 24, 2006

Phone screens, redux

It’s not so often that I read something I so totally identify with, but Joel Spolsky’s recent article on “Phone Screening” really hit the nail on the head for me as well.

I’ve just gone through a hiring process recently for my own company—several times—and finally I have made a single hire, after having gone through at least 100 résumés over the course of months of recruiting. (In a previous position, I was also required to recruit for sysadmin positions, and was frequently called in to also do interviews on developer candidates, having made a reputation for myself of being able to shake out solid candidates from weak ones.)

Like Joel, I have found that in the sheer number of applicants into the funnel requires a winnowing process, mine is several steps:

  1. I have applicants submit their information to me in a specific format. I specify in the job advertisement that their CV (or whatever) needs to be in plain text or HTML (these being formats that are amenable to being searched in a straightforward fashion in my email client). Moreover, a candidate needs to submit some kind of cover letter. Anyone who submits a Word document or PDF goes right into a folder that remains unread. Those who submit documents with an email containing a single word “here” or “I am applying for your position” gets likewise filed. I have found that this cuts out about 66% of the first group of applicants. If someone can’t be bothered to read instructions on a job page, they’re unlikely to be able to follow directions well later on, which means that they’re also unlikely to be able to think independently given customer constraints. (Three Phase Computing is a professional services company, our bread-and-butter is doing what the customer wants, and if he doesn’t know what he wants, helping him figure it out.)
  2. I then go through a phone screen with the applicant. I used to have a nice standard list of questions that I would ask candidates, but I had to revise that strongly after recruiters started preparing the bodies they shipped to me with the answers to my frequently-asked questions. Like Joel, I have found the phone screen to be a great way of avoiding wasted time on the part of all involved.
    1. The candidate doesn’t have to get dressed up and shlep out to the office (which is somewhat off the beaten path here in NYC),
    2. I don’t have to drag other people in to help me vet the prospective, and I get the benefit of immediately seeing how much the candidate can tell me.
    3. I get to find out immediately what those little time gaps on are in the resume.
    4. I get to ask a standard set of field-of-knowledge questions. I use these not so much to weed a candidate out (and I am forthright in telling them this up front, so as not to make them too nervous) but merely to find out what skills a candidate brings immediately—in other words, will the candidate hit the ground running, walking, or with a splat! (I will learn about how well he or she will continue moving in the next series of questions.)
    5. The most important things I find out, however, are how much of what the candidate puts on the CV actually matches what he or she possesses between the ears. Quite frequently I’ll see a CV littered with a line item like “Technologies:” followed by a stream of three- and four-letter acronyms. So I’ll ask them a question like “Oh, I see you list here that you know LDAP. What is that, after all?” (Yes, I have had candidates try to search on Google during a phone screen; and you can’t mask all sounds over a telephone.) I’ll get to find out if they know more than what the acronym expands into. “Where did you use it? What do you use that type of technology for? Where did you deploy it? What problems did you encounter using it?” If the candidate can give acceptable answers to this question, that’s an excellent sign!
    6. After that, I work back in reverse chronological order over the employment history, asking them specific questions about what they wrote. (I almost never go more than 3 years back, since it is unlikely that they’re going to be able to speak well about things that far distant, and it isn’t likely that anything further back than that is likely to apply immediately to any current problem. If he or she wants to talk about it, however, or bring it up as “well at XXX place we did YYY”, I’m happy to listen.) If they wrote, “deployed ZZZ technology”, I ask things like, “Well, what does that mean? Did you just install a server? Were you in charge of the project, managing others, or did you do the down-and-dirty work yourself? What did you learn from that experience?” Typically after one or two questions like that, I’ll have all the information I need to make a decision as to whether or not this candidate merits an in-person interview.
    7. After all this, I’ll ask the candidate if he or she has any questions to ask me. (I excuse them if they don’t know much; places like Craigslist will mask your identity for you, and that works well for me. I understand Joel’s job board requires people to take off their masks, and maybe for my next opening I’ll go that route.) and I explain who we are, what we do, and where I want to take the company.
    By this point, I have knocked at least another 75% off the remaining; we now have gone to a less than one-in-ten yield on our advertisement.
  3. If the candidate makes it this far, I call him or her in for an in-person interview. At the in-person, I’ll usually ask some more technical questions—more exposition on things I missed before, and maybe a few of my favorite questions. (I like to ask some SQL questions in particular about Cartesian products, after having one hapless developer do something like

    SELECT * FROM A,B;

    where tables A and B had over a million rows; I’m just glad that that query was run on the test database, and not the production one...

    In addition, I like to ask a few questions about the things we currently do. Usually I’ll take a problem that we just solved, and ask the candidate how he or she would have solved the same problem. (This is one of the best ways, I have found, of determining how a candidate actually thinks.) In addition, I’ll usually have the candidate write a small script just to see how fluent he or she is with the tools we use every day: this is mostly a follow-up to the “can he/she hit the ground running or splatting” question. Also, since the office we work in currently is small, it gives the candidate a good idea of where he or she would be working, and trying out the daily commute.

(In regards that last part: as luck would have it, our newest employee has been spending most of her time out at client sites in the three weeks since she started. She barely has really gotten to break in her new PC, although we did spend the better part of the day one day last week getting a new Cisco IP phone hooked up and talking to our asterisk install.)

August 25, 2006

Playing catchup after a brief stint away; spam, communigate and PIXen

Phew! I've spent the entire week playing catch-up.

For the first time since August 2003, I took a short vacation with my family. Just a short trip to Connecticut to see Mystic Seaport, with a short trip up to Touro Synagogue in Newport, RI,

Digression: it really is a cool synagogue; it isn’t terribly large—our synagogue here in Brooklyn is about the same size (maybe a little larger) but it wasn't built in the 1700’s, and doesn’t have its balcony finished yet. Plus, they have a 500+ year old Torah scroll given to them as a gift from the Jewish community of Amsterdam that is displayed quite nicely in a glass case....
and a trip back on a ferry from New London, CT to Orient Point, NY.


On Sunday night, I got a frantic call from a client who just converted over to using CommuniGate Pro for their mail server. It appears that the stock spam filters that they provide just don't cut the mustard. Luckily, one of my cohorts found a spamassassin conduit for CGP that appears to have stemmed the onslaught of unsolicited mail. Of course, once that was working, it uncovered yet another problem, having to do with the fact that some email from one machine wasn't making it from a qmail install on one branch of a firewall arm to another, exacerbated by the fact that I have not yet set up separate bind views , and that there is NATing going on to allow external hosts to reach the CGP machine. (The solution to that is to use qmail’s smtproutes function to point to an internal address for the CGP machine.)

Now I have to find the time to begin the architecture work for my most recent project, LTR.com, a new-and-improved dating website being started up by my acquaintance David Siegel, which I’ve put off all week...

whine whine whine

August 9, 2006

interesting mod_rewrite tricks for zencart

So I receive a request from one of my co-workers to do the following:

Please redirect https://xxx.yyy.org/index.php?main_page=product_info&products_id=21 https://xxx.yyy.org/index.php?main_page=product_info&cPath=8&products_id=21 and https://xxx.yyy.org/index.php?main_page=product_info&products_id=138 https://xxx.yyy.org/index.php?main_page=product_info&cPath=8&products_id=138 to

http://www.zzz.com/abcdefgh/nnnnn.htm


Now, the site that's being rewritten from is a big ol’ PHP application (Zencart) and it’s not really amenable to just going in and toying with—it has a nice administrative interface, but nothing obvious that lets you do a redirect from a random item in the inventory.

“Ah”, I say, “time for mod_rewrite, Apache's answer to everything.”

Well, it’s not that simple, you see.

You might think a simple RewriteRule would suffice, rewriting the whole URL at once:

RewriteRule /^index\.php\?main_page=product_info&products_id=21$ http://www.zzz.com/abcdefgh/nnnnn.htm [R]
But that won’t work, because RewriteRule only works on the URL. In this case, the URL is just https://xxx.yyy.org/index.php, and we don’t want to rewrite THAT, because, well, there are other things on the site they’d still like to sell. Remember—the trick is to match on the stuff after the question mark, and that isn't accessible to RewriteRule!

It is accessible, however, via the variable %{QUERY_STRING}. (This took me half an hour of looking and finding.) That is because the query part (that important stuff after the question mark) gets handled differently when the web server receives and parses the URL: the expectation is that whatever the URL indicates, it is something that consumes input to generate output (so it isn’t a static page, generally speaking).
In this case, RewriteCond is the thing that does the trick. I have to chain four of them together to get them all to work, but the following did the trick:

RewriteCond %{QUERY_STRING} ^main_page=product_info&products_id=21$ [OR]
RewriteCond %{QUERY_STRING} ^main_page=product_info&cPath=8&products_id=21$ [OR]
RewriteCond %{QUERY_STRING} ^main_page=product_info&products_id=138$ [OR]
RewriteCond %{QUERY_STRING} ^main_page=product_info&cPath=8&products_id=138$

RewriteRule ^/index\.php http://www.zzz.com/abcdefgh/nnnnn.htm [R=permanent]


The [OR]s at the end of the first three lines mean “combine this rule with the next one”, the RewriteRule then says “for the URIs beginning with index.php, that have already matched the conditions above me, rewrite the URL to be the new, foreign one, using an HTTP redirect with a ‘moved permanently’ condition (HTTP response code 301)”.

This appears to have done the trick, and now everyone is happier.

OK, so it isn’t heavy wizardry, but it might be useful elsewhere.

August 8, 2006

Veotag picking up steam

If you haven't tried the veotag service, give it a swing. Its secret sauce—making parts of a video searchable (so it isn’t just text tagging)—in addition to “marking up video”, is something really cool. It’s been picked up in Guy Kawasaki’s blog as well. (Guy is a multiple techie-author and big-time blogger, so this is a nice coup.)

I really didn't gush about it enough in the comments there; it's truly a cool service.

DISCLAIMER: I have no financial interest in Veotag (in case you were wondering), I’m just a satisfied user and benefit from the works some others have posted on the site.

July 25, 2006

INND and crashing machines; what you don’t get at first gloss

After the past few days of multiple outages at one of my customer's LA datacenters, I got to learn a few things about the resiliency of popular packaged unix software. Read this all the way, since the very last step is the one that came up to bite me.

innd, the popular NNTP package, does not like it when it is shut down uncleanly, e.g. by a power failure. One of the popular messages you may receive is:

Server throttled File exists writing SMstore file — throttling

What this essentially means is that the history file—the database that tells innd what articles it has seen already—has been corrupted, and the canonical thing to do (as detailed in web pages) is to rebuild your history files:
# cd /var/lib/news
# rm history*
# sudo -u news /usr/lib/news/bin/makehistory -b -O -F

...and wait for a long while, makehistory scans all the articles in your news spool to rebuild its database, and the sudo -u news makes sure that the history file has the correct ownership. (The -b flag means “delete ‘bad’ articles”, the -O means “regenerate the overview file”, and the -F means “fork off a separate process to flush the overview data to disk”.) After this, you must convert these into binary database files in dbz format:
# sudo -u news /usr/lib/news/bin/makedbz -s \
`wc -l history | awk '{print $1}'` -o

(The -s means “this is the approximate initial number of entries to use” and the -o means “don’t make any symbolic links and you copy them over manually; go ahead and overwrite old databases”.) All of that is meant to appear on one line, the \ means “continue onto the next line”.
Then you can restart innd, using your favorite method. For the Linux variety we use, it is:
# /etc/init.d/innd restart



Up to this point is the canon; it’s what you see when you google the phrase Server throttled File exists writing SMstore file — throttling.
If you go to the one Spanish-language blog describing this, you will find an additonal step: renumbering the articles that may be there.
You see, when you’ve deleted gone through the process of regenerating your history databases, you might see some (what must be, I can’t verify this) inconsistencies in the article numbers, or a difference of opinion as to what the next article number should be. Therefore, a crucial step before you start receiving new articles, but after you’ve started innd, is
# /usr/lib/news/bin/ctlinnd renumber ''

This tells the running innd that it needs to perform a scan of the overview database to make sure that it knows what number (filename) to give the next article, otherwise you can end up with errors of the sort:
Jul 24 17:34:36 HOSTNAME innd: SERVER↵
cant store article: File exists
Jul 24 17:34:36 HOSTNAME innd: tradspool: ↵
could not open /var/spool/news/articles/news/group/name/267106↵
File exists

In other words: the history database, the overview database, and the file system all need to be in agreement as to what articles have been seen, and what to name the next file. This is assuming you’re using the tradspool method of storage (newsgroup hierarchy name gets translated into unix directory hierarchy, and the article number is the file name). If you’re running some other version of file storage, you’ll need to read someone else’s blog...

July 24, 2006

who needs coffee, when you’ve got power outages

So I get to my office this morning and find multiple calls from my customer who has a data center in LA. Evidently they still have power problems in LA this morning, because when I call up the NOC guy in our data center, he says that he’s been up for 48 hours dealing with this.

Still: where are your battery backups? Where are your diesel gensets?

Now I get to rebuild history files again, figure out why MySQL replication isn’t automatically reconnecting (I think I know why now, though, and it smells like pilot error...) and watch as the load on our master servers jumps well into the double-digits due to the inrush of MySQL replicants and file replications...



In other news, I did find a really cool firefox extension called Scrapbook, which allows you to save copies of web pages, annotate them, and even edit them before and after saving. It’s the answer to another customer’s desires, who loves to save web pages “as they are”. It’s also cool to save amusing Craigslist ads for posterity. (For the record, that particular ad, which will expire in about 20 days, was a paid ad. And, of course, I’ve saved it.)

July 23, 2006

Lessons learned after a major system crash

I wanted to title this something snide about co-location clowns again, but I won’t. At this hour, the anger won’t do any good. My apologies if this isn’t as coherent as it could be.

This evening, one of my client’s data centers had a major power outage. (No, they’re not in Queens, NYC.) I found out about it right after the Sabbath ended by a phone call from one of my client’s clients, whose own monitoring was going bananas—it happened in between the Saturday coverage’s most recent check and my first check post-Sabbath.
(Yes, they have UPSes...allegedly. No, I do not know why the UPSs did not kick in. We’re waiting to hear back from them for a RCA [root cause analysis] to determine what needs to be done.)

After a major outage, you learn a lot of things about your system:

Continue reading "Lessons learned after a major system crash" »

July 19, 2006

Asterisk on FreeBSD, the saga begins

(Grr. I lost the first edit of this.)

Well, I managed to get Asterisk running up on my FreeBSD 6.1 machine. A few things I've learned along the way:

  1. The book Asterisk: The Future of Telephony is an excellent starting reference. (I get no kickbacks from the link.)
  2. However, don’t ignore the sample configurations: they are extensively commented, and very useful.
  3. The order of statements in certain stanzas of certain files (zapata.conf) is more crucial than the book or the sample files may lead you to believe. Yes, of course the order in the dialplan is crucial, but we knew that coming in. I'm talking about the order of the zaptel card configuration stuff.
  4. The out-of-the-box installation of many parts of Asterisk—voicemail for one—is just plain fantastic. (The dial-by-name directory is also excellent.)
  5. There appear to be hardware issues between my Zaptel card and my on-board ethernet (which I have mentioned previously)
Other than the hardware issues, my little PBX works just fine. I managed to record a custom opening menu as well, using templates provided in the book above. Oddly enough, the voice on the opening menu sounds strangely like that baritone voice at Omnipod’s main number at +1 212 620 2845. I'm surprised that they’ve never updated that. Heck, they never even got rid of my old voicemail message at my old extension, and for almost a full year after I left, I still received phone calls on my cellphone, since option 2 of my extention calls my cellphone, and my name still appears in the dial-by-name directory there. Go figure.

I suppose I should find someone whose voice is a little bit higher in tenor for better audibility, too.

All in all, though, setting up the entire thing without missteps probably only took about 5 or 6 hours all told, which would probably have been as long as it would have taken to iron out all the kinks if I had purchased a service directly from Verizon.

Next steps are to get Mark Shoulson an IP phone in his house, so he can feel like a real Grown Up Employee of 3 Phase...

July 5, 2006

Centos vs. FreeBSD 6.1, part two

I think this time it's the eMachines machine I purchased.

After

  • getting an el-cheapo RealTek-based ethernet card (and massively overpaying for it at a storefront, but I needed it right now) since neither CentOS nor FBSD could recognize the on-board controller, for reasons unbeknownst to me, and
  • removing the modem, which I didn't need anyway,
FreeBSD 6.1 installed without a hitch (although I have not yet RAIDed my two disks. Did I mention how bad an idea it is to put two disks on the same PATA channel in a RAID pair? I evidently have not learned my own lesson yet.). I now have asterisk up and running on the machine, and my one FXO port is working just dandy (after going out and buying a $10 power-cable splitter, since the provided power cables in the machine were not long enough to reach the bottom PCI slot).

I haven't re-tried CentOS, but I am pretty sure that one of the half-dozen unrecorded changes I made did the trick.

Now, as for getting my Sipura SPA-921 phone to work correctly with Asterisk, that's for another entry...

June 15, 2006

Centos vs. Freebsd 6.1 installation travails

So the other week I begin to put together the materiel necessary to build that Asterisk box I've been needing to build—it doesn't seem fit for a company whose motto is “industrial-strength computing” to have no real fancy VoIP/messaging system. This is all a grand experiment and learning experience for me, so I fully expect a few missteps along the way. I'm therefore taking this as an opportunity to make a few mistakes along the way.

Being partial as I am to the *BSD series of Unices, I decide to download the 2 ISOs for FreeBSD 6.1, burn them to CD and begin to install.

I must give an aside about the hardware. I'm trying to build this machine on the cheap, but I do want to have it follow at least some notion of “best practices”. Anything that is remotely server-like gets two disk drives in a RAID1 array, disk drives being the #1 thing in my experience to up and fail on you (#2 being power supplies and #3 being—of all things—ethernet cards!). I notice that the local Best Buy has eMachines (yeah, I know, low-end consumer grade stuff? But I've had only good experiences with eMachines to date, and the cost of being wrong here is only about $400) on sale, and 160GB disk drives too, so I walk into the one in Staten Island, plunk down my credit card, and come out with a new eMachines T3418, a spare 160GB drive (to sort-of match the one in there), and another 17" monitor. (You have to buy the monitor in order to get the rebate, but the monitor on my sons' computer is about 10 years old and due to die, and it was basically free after rebate, so why not?)

Continue reading "Centos vs. Freebsd 6.1 installation travails" »

June 14, 2006

Followup on HP

My new printer arrived yesterday via Fedex ground, and by all appearances works fine. I packed the old one up in the box, called up UPS (it turns out for ARS packages there is no pickup fee to me — usually if I call them for a pickup, there is a pickup charge, but not for return packages! w00t) and they're coming tomorrow to cart away the carcass of my old system.

HP so far has done me right.

June 7, 2006

HP printers...2 strikes now

So yesterday my relatively new (3 months) HP Officejet 7210 all-in-one printer -- really a nice machine, comes with an ethernet interface so I could put it far away from any machine printing to it, and close to the phone jacks so it can be a good fax, flatbed scanner/copier, etc. -- decides to up and die on me, telling me that the color printer cartridge (that came with the machine!) is, all of the sudden, the incorrect cartridge for the printer.

Um, this is kind of déjà vu for me now, since my OfficeJet 4215 died a similar death back in March, which is what prompted me to spring the extra $200 on the fancier, better-dressed printer.

(Of course, it's not the $1000 laserjet machine, but I needed to buy a printer RIGHT THEN and that's what Staples had in stock that evening)


So, I go to HP's site looking for support on it, and lo and behold they give me instructions: remove and reinsert the cartidges. If that doesn't work, then “call support”. Calling in this case is using their 24/7 web-chat-support feature with someone (most likely in a timezone 12 hours from my own).

I can't complain about that, because the chat works, and the support guy helped answer my questions.

First I had to

Follow the steps below to power cycle your all-in-one: 1. Unplug the all-in-one from power and disconnect the connection port.(USB) 2. Wait 30 seconds. 3. Plug in the power only. 4. Repeat steps 2-3 two more times. 5. On the third time after plugging the unit into power, reconnect the connection port from the all-in-one to your computer.

Two to three times I have to powercycle this beastie! Sweet fancy Mushke, would it have killed them to put in a “really hard we mean it” hard reset button?

Oh well, so I jump through these hoops. (Reminds me of the old joke: how can you tell the field-service rep changing his tires on the side of the road? He's the one swapping tires in and out to see which one is flat.)

What I did not like was that part of the debugging process involves breaking into a new package of (expensive, ’natch) cartridges to try them out. But that's just me being cheap. With two new cartridges in the machine, my printer now reads:

Insert Print Cartridges
but that's neither here nor there.

After all of this, Mr. Tech Support declares:


xxxxxx: This shows the issue seems to be with the printer hardware
xxxxxx: Please let me know the serial number of the All-in-One
[my response]
xxxxxx: It appears that this device has experienced a hardware failure and I shall be glad to process the request for a printer replacement with your permission for free of cost and you will receive the unit with in 5 to 7 business days.
jerry altzman: that would be delightful

Well, raise my rent! They're going to ship me a new one. Of course, this item is a “collateral product”, and therefore


xxxxxx: However, the All-in-One is a collateral product, that is, you would be required to ship the defective All-in-One on receipt of the replacement. Therefore, you would be required to provide us the credit card information as security.

NOTE: do not provide the Credit Card information in this chat session

In cases where HP did not receive the defective part/unit within 30 calendar days of shipment of the exchange part/unit from you then you will be charged for the exchange part/unit.Return instructions and pre-paid shipping label are included.


(You can picture him cutting and pasting from his script file right into the chat window.) I get a call 20 minutes later by someone named “Archie” (surely short for Akhbar Samagutrapan or some similar indic Upper-Baluchistanian name, judging by the heavy accent on the phone) asking for my credit card number, which I provide (no, I've gone through credit card fraud once on my business card, I'm not going to provide it to you, too) and now in 5 to 7 business days they'll ship me a new machine. Of course, I'm without printer and fax machine during that time, so it's time to handwrite those paychecks.

May 22, 2006

Joe Jobbed in the Domain Name Space

Today I received an interesting letter from "Domain Registry of America" telling me that the registration on a domain of mine "stefy72.com" is about to expire.

Well gee, that's odd, I've never registered a domain like that. Moreover, it has the stinky smell of a fishing site or some such.

Continue reading "Joe Jobbed in the Domain Name Space" »

May 11, 2006

More on Hiring Geeks

Well, my ad went up 3 weeks ago (or so) and I've received a somewhat tepid response. I'm not sure if it's my tepid prose or something else, but response this time was about 53 résumés received, of which 16 went immediately into the “clown” folder for not following directions.

Some of the people obviously missed the

Unix sysadmin for 15-20 hours a week in-office

part, because I got letters that said
I have a full time position, but am looking for extra work.
or
I currently working full time and I am seeking a part time opportunity.

which, of course, is inconsistent with my request:
You must be able to think on your feet, and you must be able to read and follow directions.
(Extra emphasis added).

Now, of course, I don't have that much of a problem with people who aren't qualified (yet) but I don't like people who simply don't read the advertisement well.

I'm still looking, in case anyone cares.

April 21, 2006

Hiring geeks

(I never said my blog would be wholly noncommercial!) Yes, my company, 3 Phase Computing, is hiring (that link will expire in May). Whenever we hire, I like to provide a first-pass "clown filter" to weed out the obviously incapable. When hiring for Unix people, one way you can do that is to require that they provide their résumé or CV in PDF (OpenOffice will generate one for you); only the clueful under Windows will have a PDF generator installed. I prefer to have a line that reads:
  • ...
  • You must be able to think on your feet, and you must be able to read and follow directions.
And then require that all submissions be in plain-text or HTML format. That serves several purposes:
  1. It saves me a step when reading through the piles of incoming mail--plain text and HTML are rendered in-line in Thunderbird.
  2. It forces people to think a second before sending out their CV in the standard DOC format
  3. It makes it easier for me to search for text strings through my email using my email client
  4. Finally, it makes it plain and clear who does and does not actually read and follow directions
Other people favor requiring a bizarre subject line or text line in the body of the email (one of my clients likes doing that and it works well for him) but my method is slightly better insofar as it also serves the other purposes I listed. One of my friends (who wishes to remain anonymous) also warned me:
(19:09:30) xxxxxxxxxxx: PREPARE fo da Indians

(19:09:34) xxxxxxxxxxx: ROFL

which I had hoped to avoid by saying in my post "in-office", but alas, I probably should have said (in the great tradition of Craigslist postings in NYC) "NO ONE FROM OVERSEAS! IF YOU'RE NOT IN NYC, SORRY." Subtlety is lost in the classifieds...