My HTPC is almost appliance like, in the way I never upgrade it, i.e. this morning it was still running Ubuntu 11.04 Natty Narwhal… It’s also not very appliance like in that it’s also full of random development stuff that probably shouldn’t be on it as well as several different types of databases, my backup infrastructure and tons of other things you really don’t care for.

But not today. Today I was bored and decided the best use of my time was to replace MythTV with XBMC. Not only that, I figured I should probably upgrade from

There are several things to think about when taking control of another computer, the Operating System running on it, the speed of your network connection and the tools you have at your disposal.

How To Take Control Of Another Computer

Operating System

There are 3 main choices of Operating system that the computer you want to take control of might be running, Windows, Mac OS X and Linux. Fortunately Mac OS X is based on BSD so the tools you would use to take control of it are the same as you would for Linux, simplifying things some what!

We’re using an old version of Upstart, on Centos, to manage stopping and starting our Node.js daemons, and one of the things the script does, like any good deamon, is change the user of the deamon process from root to something more applicable, security and all that 😉

The scripts look a little like this

!upstart
description "Amazing Node.js Daemon"
author "idimmu"
start on runlevel [2345]
stop on shutdown
env PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
env NAME="amazing-daemon"
script
export HOME="/root"
cd /opt/idimmu/$NAME
echo $$ > /var/run/$NAME.pid
exec sudo -u idimmu /usr/bin/node /opt/idimmu/$NAME/server.js >> /var/log/$NAME/stdout.log 2>&1
end script
pre-start script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (upstart) Starting $NAME" >> /var/log/$NAME/stdout.log
end script
pre-stop script
rm /var/run/$NAME.pid
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (upstart) Stopping $NAME" >> /var/log/$NAME/stdout.log
end script

Which is nice, as it means we can use Upstart to stop/start/status deamons really nicely. The equivalent init.d script looked really horrible.

I’ve got a rather large dataset that I need to do a lot of processing on, over several iterations, it’s a 20gb zip file, flat text, and I’m impatient and don’t like not knowing things!

My new favourite Linux command line tool, pv (pipe viewer) is totally awesome. Check this out:

<br /> pv -cN source < urls.gz | zcat | pv -cN zcat | perl -lne '($a,$b,$c,$d) = split /\||\t/; print $b unless $b =~ /ac\.uk/; print $c unless $c =~ /ac\.uk/' | pv -cN perl | gzip | pv -cN gzip > hosts.gz<br /> zcat: 93.4GiB 1:33:18 [26.6MiB/s] [ <=> ]<br /> perl: 85.7GiB 1:33:18 [25.3MiB/s] [ <=> ]<br /> source: 13.2GiB 1:33:17 [3.57MiB/s] [===============================================> ] 67% ETA 0:44:41<br /> gzip: 12.7GiB 1:33:18 [3.51MiB/s] [ <=> ]<br />

Whilst working an AMAZING NPM repository mirror yesterday (which totally works, despite not really offering the performance benefit I’d hoped, because NPM is rubbish) I came across this error whilst doing things

<br /> 16 http GET https://localhost:5984/registry/_design/app/_rewrite/-/all/since?stale=update_after&startkey=1371737164294<br /> 17 http 500 https://localhost:5984/registry/_design/app/_rewrite/-/all/since?stale=update_after&startkey=1371737164294<br /> 18 error Error: insecure_rewrite_rule too many ../.. segments: registry/_design/app/_rewrite/-/all/since<br /> 18 error at RegClient.<anonymous> (/root/.nvm/v0.8.15/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:259:14)<br /> 18 error at Request.init.self.callback (/root/.nvm/v0.8.15/lib/node_modules/npm/node_modules/request/main.js:120:22)<br /> 18 error at Request.EventEmitter.emit (events.js:99:17)<br /> 18 error at Request.<anonymous> (/root/.nvm/v0.8.15/lib/node_modules/npm/node_modules/request/main.js:648:16)<br /> 18 error at Request.EventEmitter.emit (events.js:126:20)<br /> 18 error at IncomingMessage.Request.start.self.req.self.httpModule.request.buffer (/root/.nvm/v0.8.15/lib/node_modules/npm/node_modules/request/main.js:610:14)<br /> 18 error at IncomingMessage.EventEmitter.emit (events.js:126:20)<br /> 18 error at IncomingMessage._emitEnd (http.js:366:10)<br /> 18 error at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:149:23)<br /> 18 error at Socket.socketOnData [as ondata] (http.js:1367:20)<br /> 19 error If you need help, you may report this log at:<br /> 19 error <https://github.com/isaacs/npm/issues><br /> 19 error or email it to:<br /> 19 error <npm-@googlegroups.com><br />

We use Node.js a LOT, which means we do npm install a LOT. And npm is pretty terrible, with horrible dependency handling so we can end up requesting hundreds of dependent modules with it’s recursive patten e.g. for just one of our projects we can end up with paths like

`
./node_modules/bcrypt/node_modules/nodeunit/node_modules/should/node_modules/mocha/node_modules/glob/node_modules/minimatch/node_modules/sigmund/node_modules/tap/node_modules

[root@hmon workspace]# find . -name node_modules | wc -l
2103
`

That’s 2103 node_modules directories, for an application we’ve written that has only 22 dependencies configured for it!