How To Create An NPM Repository Mirror

How To Create An NPM Repository Mirror

npmWe use Node.js a LOT, which means we do npm install a LOT. And npm is pretty terrible, with horrible dependency handling so we can end up requesting hundreds of dependent modules with it’s recursive patten e.g. for just one of our projects we can end up with paths like


./node_modules/bcrypt/node_modules/nodeunit/node_modules/should/node_modules/mocha/node_modules/glob/node_modules/minimatch/node_modules/sigmund/node_modules/tap/node_modules

[[email protected] workspace]# find . -name node_modules | wc -l
2103

That’s 2103 node_modules directories, for an application we’ve written that has only 22 dependencies configured for it!


[[email protected] workspace]# find . -name mocha | wc -l
59

There are 59 instances of the mocha module in the dependency chain, how is that for terrible reuse of code! Why can’t npm be nice like every other language out there, e.g. perl (hi cpan), PHP, Ruby (hi gems!) and Python??

npm does cache locally, but it kind of sucks.

Anyway, rant over, we want to create a mirror of the npm repository to mitigate periods of npm outages (occasionally it does have them) and hopefully speed things up a little bit, so here’s how I did it!

CouchDB

All the NPM data is stored in couchdb, I’m doing this on Centos so I’m going to use yum to install couchdb


[[email protected] etc]# yum install couchdb
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.mirror.linuxwerk.com
* epel: mirrors.n-ix.net
* extras: centos.mirror.linuxwerk.com
* passenger: passenger.stealthymonkeys.com
* rpmforge: mirror1.hs-esslingen.de
* rpmforge-extras: mirror1.hs-esslingen.de
* rpmforge-testing: mirror1.hs-esslingen.de
* updates: mirror.optimate-server.de
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package couchdb.x86_64 0:1.2.1-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================================================================================
Package Arch Version Repository Size
=================================================================================================================================================================================================================
Installing:
couchdb x86_64 1.2.1-1 drum 1.1 M

Transaction Summary
=================================================================================================================================================================================================================
Install 1 Package(s)

Total download size: 1.1 M
Installed size: 3.0 M
Is this ok [y/N]: y
Downloading Packages:
couchdb-1.2.1-1.x86_64.rpm | 1.1 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : couchdb-1.2.1-1.x86_64 1/1
Verifying : couchdb-1.2.1-1.x86_64 1/1

Installed:
couchdb.x86_64 0:1.2.1-1

Complete!

Simples! Next step is to start it, confirm it’s listening on a port and test it works!


[[email protected] etc]# /etc/init.d/couchdb start
Starting database server couchdb
[[email protected] etc]# ps aux | grep couch
root 9790 0.0 0.0 106188 1532 pts/1 S 14:41 0:00 /bin/sh -e /opt/netdev/erlang/bin/couchdb -a /etc/couchdb/default.ini -a /etc/couchdb/local.ini -b -r 0 -p /var/run/couchdb/couchdb.pid -o couchdb.stdout -e couchdb.stderr -R
root 9800 0.0 0.0 106188 760 pts/1 S 14:41 0:00 /bin/sh -e /opt/netdev/erlang/bin/couchdb -a /etc/couchdb/default.ini -a /etc/couchdb/local.ini -b -r 0 -p /var/run/couchdb/couchdb.pid -o couchdb.stdout -e couchdb.stderr -R
root 9801 0.7 0.1 666732 18576 pts/1 Sl 14:41 0:00 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib64/erlang -progname erl -- -home /root -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch -pidfile /var/run/couchdb/couchdb.pid -heart
root 9834 0.0 0.0 103236 872 pts/1 S+ 14:42 0:00 grep couch
root 26078 0.0 0.0 173292 1720 ? S May07 0:00 sudo -u drum setsid node /opt/netdev/drum-collab-provisioning/server/server-couch.js
drum 26079 0.0 0.4 999360 70064 ? Ssl May07 18:09 node /opt/netdev/drum-collab-provisioning/server/server-couch.js
[[email protected] etc]# netstat -lpn | grep 9801
tcp 0 0 127.0.0.1:5984 0.0.0.0:* LISTEN 9801/beam.smp

At the moment in it’s default configuration it’s only listening on 127.0.0.1 so we want to fix that!

We also want to ensure that secure_rewrites is disabled else NPM will spit out loads of errors and not work!

In /etc/couchdb/local.ini we can override the default configuration, so we want to change


[httpd]
;port = 5984
;bind_address = 127.0.0.1

to


[httpd]
;port = 5984
bind_address = 0.0.0.0
secure_rewrites = false

and then restart couchdb with /etc/init.d/couchdb restart! Now we can see couchdb listening on all ports and test it with curl


[[email protected] couchdb]# netstat -lpn | grep 5984
tcp 0 0 0.0.0.0:5984 0.0.0.0:* LISTEN 10011/beam.smp
[[email protected] couchdb]# curl https://localhost:5984
{"couchdb":"Welcome","version":"1.2.1"}

Yay, we’re alive!!!

Setting Up CouchDB Replication

Now we need to tell couchdb that it needs to replicate from the NPM master in a continuous fashion, so as the NPM master updates, so does our couchdb instance!


[[email protected] couchdb]# curl -X POST https://127.0.0.1:5984/_replicate -d '{"source":"https://isaacs.iriscouch.com/registry/", "target":"registry", "continuous":true, "create_target":true}' -H "Content-Type: application/json"
{"ok":true,"_local_id":"d0818878c462afa6791440ab08348394+continuous+create_target"}

And we’re off! You can interrogate how the replcation is doing by visiting the server with a webbrowser at https://hostname:5984/_utils/ it should look a little like this:

couchdb-npm-mirror

Eventually it will stop growing, I promise 😉 As of writing it’s just shy of 50GB

NPM Repository Mirror

Configuring NPM To Use Your Mirror

First we need to install some random npmjs stuff in to our couch database


git clone git://github.com/isaacs/npmjs.org.git
cd npmjs.org
sudo npm install -g couchapp
npm install couchapp
npm install semver
couchapp push registry/app.js https://localhost:5984/registry
couchapp push www/app.js https://localhost:5984/registry

Which looks a little bit like this


[[email protected] ~]# git clone git://github.com/isaacs/npmjs.org.git
Initialized empty Git repository in /root/npmjs.org/.git/
remote: Counting objects: 1291, done.
remote: Compressing objects: 100% (742/742), done.
remote: Total 1291 (delta 609), reused 1200 (delta 531)
Receiving objects: 100% (1291/1291), 649.70 KiB | 353 KiB/s, done.
Resolving deltas: 100% (609/609), done.
[[email protected] ~]# cd npmjs.org
[[email protected] npmjs.org]# npm install -g couchapp
npm http GET https://registry.npmjs.org/couchapp
npm http 304 https://registry.npmjs.org/couchapp
npm http GET https://registry.npmjs.org/watch
npm http GET https://registry.npmjs.org/request
npm http 304 https://registry.npmjs.org/request
npm http 304 https://registry.npmjs.org/watch
npm http GET https://registry.npmjs.org/qs
npm http GET https://registry.npmjs.org/json-stringify-safe
npm http GET https://registry.npmjs.org/forever-agent
npm http GET https://registry.npmjs.org/tunnel-agent
npm http GET https://registry.npmjs.org/http-signature
npm http GET https://registry.npmjs.org/hawk
npm http GET https://registry.npmjs.org/aws-sign
npm http GET https://registry.npmjs.org/oauth-sign
npm http GET https://registry.npmjs.org/cookie-jar
npm http GET https://registry.npmjs.org/node-uuid
npm http GET https://registry.npmjs.org/mime
npm http GET https://registry.npmjs.org/form-data/0.0.8
npm http 200 https://registry.npmjs.org/json-stringify-safe
npm http GET https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-4.0.0.tgz
npm http 200 https://registry.npmjs.org/forever-agent
npm http GET https://registry.npmjs.org/forever-agent/-/forever-agent-0.5.0.tgz
npm http 200 https://registry.npmjs.org/http-signature
npm http 200 https://registry.npmjs.org/tunnel-agent
npm http GET https://registry.npmjs.org/http-signature/-/http-signature-0.9.11.tgz
npm http GET https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.3.0.tgz
npm http 200 https://registry.npmjs.org/qs
npm http GET https://registry.npmjs.org/qs/-/qs-0.6.5.tgz
npm http 200 https://registry.npmjs.org/aws-sign
npm http GET https://registry.npmjs.org/aws-sign/-/aws-sign-0.3.0.tgz
npm http 200 https://registry.npmjs.org/oauth-sign
npm http GET https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.3.0.tgz
npm http 200 https://registry.npmjs.org/cookie-jar
npm http GET https://registry.npmjs.org/cookie-jar/-/cookie-jar-0.3.0.tgz
npm http 200 https://registry.npmjs.org/node-uuid
npm http GET https://registry.npmjs.org/node-uuid/-/node-uuid-1.4.0.tgz
npm http 200 https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-4.0.0.tgz
npm http 200 https://registry.npmjs.org/forever-agent/-/forever-agent-0.5.0.tgz
npm http 200 https://registry.npmjs.org/mime
npm http GET https://registry.npmjs.org/mime/-/mime-1.2.9.tgz
npm http 200 https://registry.npmjs.org/form-data/0.0.8
npm http GET https://registry.npmjs.org/form-data/-/form-data-0.0.8.tgz
npm http 200 https://registry.npmjs.org/http-signature/-/http-signature-0.9.11.tgz
npm http 200 https://registry.npmjs.org/qs/-/qs-0.6.5.tgz
npm http 200 https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.3.0.tgz
npm http 200 https://registry.npmjs.org/aws-sign/-/aws-sign-0.3.0.tgz
npm http 200 https://registry.npmjs.org/cookie-jar/-/cookie-jar-0.3.0.tgz
npm http 200 https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.3.0.tgz
npm http 200 https://registry.npmjs.org/node-uuid/-/node-uuid-1.4.0.tgz
npm http 200 https://registry.npmjs.org/mime/-/mime-1.2.9.tgz
npm http 200 https://registry.npmjs.org/form-data/-/form-data-0.0.8.tgz
npm http 200 https://registry.npmjs.org/hawk
npm http GET https://registry.npmjs.org/hawk/-/hawk-0.13.1.tgz
npm http 200 https://registry.npmjs.org/hawk/-/hawk-0.13.1.tgz
npm http GET https://registry.npmjs.org/assert-plus/0.1.2
npm http GET https://registry.npmjs.org/asn1/0.1.11
npm http GET https://registry.npmjs.org/ctype/0.5.2
npm http GET https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/boom
npm http GET https://registry.npmjs.org/cryptiles
npm http GET https://registry.npmjs.org/sntp
npm http GET https://registry.npmjs.org/combined-stream
npm http GET https://registry.npmjs.org/async
npm http 200 https://registry.npmjs.org/ctype/0.5.2
npm http 200 https://registry.npmjs.org/assert-plus/0.1.2
npm http 200 https://registry.npmjs.org/asn1/0.1.11
npm http 200 https://registry.npmjs.org/boom
npm http GET https://registry.npmjs.org/ctype/-/ctype-0.5.2.tgz
npm http GET https://registry.npmjs.org/boom/-/boom-0.4.2.tgz
npm http GET https://registry.npmjs.org/assert-plus/-/assert-plus-0.1.2.tgz
npm http GET https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz
npm http 200 https://registry.npmjs.org/cryptiles
npm http 200 https://registry.npmjs.org/combined-stream
npm http GET https://registry.npmjs.org/cryptiles/-/cryptiles-0.2.1.tgz
npm http GET https://registry.npmjs.org/combined-stream/-/combined-stream-0.0.4.tgz
npm http 200 https://registry.npmjs.org/sntp
npm http 200 https://registry.npmjs.org/ctype/-/ctype-0.5.2.tgz
npm http 200 https://registry.npmjs.org/boom/-/boom-0.4.2.tgz
npm http GET https://registry.npmjs.org/sntp/-/sntp-0.2.4.tgz
npm http 200 https://registry.npmjs.org/assert-plus/-/assert-plus-0.1.2.tgz
npm http 200 https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz
npm http 200 https://registry.npmjs.org/cryptiles/-/cryptiles-0.2.1.tgz
npm http 200 https://registry.npmjs.org/combined-stream/-/combined-stream-0.0.4.tgz
npm http 200 https://registry.npmjs.org/sntp/-/sntp-0.2.4.tgz
npm http 200 https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/hoek/-/hoek-0.8.5.tgz
npm http 200 https://registry.npmjs.org/async
npm http 200 https://registry.npmjs.org/hoek/-/hoek-0.8.5.tgz
npm http GET https://registry.npmjs.org/async/-/async-0.2.9.tgz
npm http GET https://registry.npmjs.org/hoek
npm http 200 https://registry.npmjs.org/async/-/async-0.2.9.tgz
npm http GET https://registry.npmjs.org/delayed-stream/0.0.5
npm http 304 https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/hoek/-/hoek-0.9.1.tgz
npm http 200 https://registry.npmjs.org/delayed-stream/0.0.5
npm http GET https://registry.npmjs.org/delayed-stream/-/delayed-stream-0.0.5.tgz
npm http 200 https://registry.npmjs.org/hoek/-/hoek-0.9.1.tgz
npm http 200 https://registry.npmjs.org/delayed-stream/-/delayed-stream-0.0.5.tgz
/root/.nvm/v0.8.15/bin/couchapp -> /root/.nvm/v0.8.15/lib/node_modules/couchapp/bin.js
[email protected] /root/.nvm/v0.8.15/lib/node_modules/couchapp
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
[[email protected] npmjs.org]# npm install couchapp
npm http GET https://registry.npmjs.org/couchapp
npm http 304 https://registry.npmjs.org/couchapp
npm http GET https://registry.npmjs.org/watch
npm http GET https://registry.npmjs.org/request
npm http 304 https://registry.npmjs.org/request
npm http 304 https://registry.npmjs.org/watch
npm http GET https://registry.npmjs.org/qs
npm http GET https://registry.npmjs.org/json-stringify-safe
npm http GET https://registry.npmjs.org/forever-agent
npm http GET https://registry.npmjs.org/tunnel-agent
npm http GET https://registry.npmjs.org/http-signature
npm http GET https://registry.npmjs.org/hawk
npm http GET https://registry.npmjs.org/aws-sign
npm http GET https://registry.npmjs.org/oauth-sign
npm http GET https://registry.npmjs.org/cookie-jar
npm http GET https://registry.npmjs.org/node-uuid
npm http GET https://registry.npmjs.org/mime
npm http GET https://registry.npmjs.org/form-data/0.0.8
npm http 304 https://registry.npmjs.org/json-stringify-safe
npm http 304 https://registry.npmjs.org/qs
npm http 304 https://registry.npmjs.org/forever-agent
npm http 304 https://registry.npmjs.org/tunnel-agent
npm http 304 https://registry.npmjs.org/http-signature
npm http 304 https://registry.npmjs.org/hawk
npm http 304 https://registry.npmjs.org/aws-sign
npm http 304 https://registry.npmjs.org/oauth-sign
npm http 304 https://registry.npmjs.org/cookie-jar
npm http 304 https://registry.npmjs.org/node-uuid
npm http 304 https://registry.npmjs.org/mime
npm http 304 https://registry.npmjs.org/form-data/0.0.8
npm http GET https://registry.npmjs.org/assert-plus/0.1.2
npm http GET https://registry.npmjs.org/asn1/0.1.11
npm http GET https://registry.npmjs.org/ctype/0.5.2
npm http GET https://registry.npmjs.org/boom
npm http GET https://registry.npmjs.org/cryptiles
npm http GET https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/sntp
npm http GET https://registry.npmjs.org/combined-stream
npm http GET https://registry.npmjs.org/async
npm http 304 https://registry.npmjs.org/ctype/0.5.2
npm http 304 https://registry.npmjs.org/asn1/0.1.11
npm http 304 https://registry.npmjs.org/assert-plus/0.1.2
npm http 304 https://registry.npmjs.org/boom
npm http 304 https://registry.npmjs.org/cryptiles
npm http 304 https://registry.npmjs.org/hoek
npm http 304 https://registry.npmjs.org/sntp
npm http 304 https://registry.npmjs.org/combined-stream
npm http 304 https://registry.npmjs.org/async
npm http GET https://registry.npmjs.org/hoek
npm http GET https://registry.npmjs.org/delayed-stream/0.0.5
npm http 304 https://registry.npmjs.org/hoek
npm http 304 https://registry.npmjs.org/delayed-stream/0.0.5
[email protected] node_modules/couchapp
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
[[email protected] npmjs.org]# npm install semver
npm http GET https://registry.npmjs.org/semver
npm http 200 https://registry.npmjs.org/semver
npm http GET https://registry.npmjs.org/semver/-/semver-1.0.14.tgz
npm http 200 https://registry.npmjs.org/semver/-/semver-1.0.14.tgz
[email protected] node_modules/semver
[[email protected] npmjs.org]# couchapp push registry/app.js https://localhost:5984/registry
Preparing.
Serializing.
PUT https://localhost:5984/registry/_design/scratch
Finished push. 1-8b4b7cde241179296b34d437d6fcbec3
[[email protected] npmjs.org]# couchapp push www/app.js https://localhost:5984/registry
Preparing.
Serializing.
PUT https://localhost:5984/registry/_design/ui
Finished push. 1-7d950267677c7a20ec944e4c8385af2f

Ok, now we should in theory have a completly working mirror repository of npm now!

Testing Your NPM Repository Mirror

Testing the npm repo mirror is easy enough, first we’ll make a request to the official repo then compare it to our server!


[[email protected]on npmjs.org]# npm search cakes
NAME DESCRIPTION AUTHOR DATE KEYWORDS
mocha-cakes bdd stories add-on for mocha test framework with cucumber given/then/when syntax. =quangv 2012-06-03 15:30 mocha bdd stories cucumber test testing gherkin acceptance customer functional end-user
npm http GET https://registry.npmjs.org/-/all/since?stale=update_after&startkey=1371737099041
npm http 200 https://registry.npmjs.org/-/all/since?stale=update_after&startkey=1371737099041

Ok the official mirror has something called mocha-cakes in it, sounds delicious!


[[email protected] couchdb]# npm search cakes
npm http GET https://localhost:5984/registry/_design/app/_rewrite/-/all/since?stale=update_after&startkey=1371802019107
npm http 200 https://localhost:5984/registry/_design/app/_rewrite/-/all/since?stale=update_after&startkey=1371802019107
NAME DESCRIPTION AUTHOR DATE KEYWORDS
mocha-cakes bdd stories add-on for mocha test framework with cucumber given/then/when syntax. =quangv 2012-06-03 15:30 mocha bdd stories cucumber test testing gherkin acceptance customer functional end-user

And so does ours!!! It’s still syncing so I’m going to forgive that 404

Telling NPM To Use Our NPM Repository Mirror

We can either configure ~/.npmrc with


registry = https://localhost:5984/registry/_design/app/_rewrite

or set it on the command line with


npm config set registry https://localhost:5984/registry/_design/app/_rewrite

Bamo, a dozen commands later and now you too have your own working npm repository 😀