How To Create An NPM Repository Mirror

Posted on
How To Create An NPM Repository Mirror

npmWe use Node.js a LOT, which means we do npm install a LOT. And npm is pretty terrible, with horrible dependency handling so we can end up requesting hundreds of dependent modules with it’s recursive patten e.g. for just one of our projects we can end up with paths like

`
./node_modules/bcrypt/node_modules/nodeunit/node_modules/should/node_modules/mocha/node_modules/glob/node_modules/minimatch/node_modules/sigmund/node_modules/tap/node_modules

[root@hmon workspace]# find . -name node_modules | wc -l
2103
`

That’s 2103 node_modules directories, for an application we’ve written that has only 22 dependencies configured for it!

<br /> [root@hmon workspace]# find . -name mocha | wc -l<br /> 59<br />

There are 59 instances of the mocha module in the dependency chain, how is that for terrible reuse of code! Why can’t npm be nice like every other language out there, e.g. perl (hi cpan), PHP, Ruby (hi gems!) and Python??

npm does cache locally, but it kind of sucks.

Anyway, rant over, we want to create a mirror of the npm repository to mitigate periods of npm outages (occasionally it does have them) and hopefully speed things up a little bit, so here’s how I did it!

CouchDB

All the NPM data is stored in couchdb, I’m doing this on Centos so I’m going to use yum to install couchdb

`
[root@hmon etc]# yum install couchdb
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

  • base: centos.mirror.linuxwerk.com
  • epel: mirrors.n-ix.net
  • extras: centos.mirror.linuxwerk.com
  • passenger: passenger.stealthymonkeys.com
  • rpmforge: mirror1.hs-esslingen.de
  • rpmforge-extras: mirror1.hs-esslingen.de
  • rpmforge-testing: mirror1.hs-esslingen.de
  • updates: mirror.optimate-server.de
    Setting up Install Process
    Resolving Dependencies
    –> Running transaction check
    —> Package couchdb.x86_64 0:1.2.1-1 will be installed
    –> Finished Dependency Resolution

Dependencies Resolved

=================================================================================================================================================================================================================
Package Arch Version Repository Size
=================================================================================================================================================================================================================
Installing:
couchdb x86_64 1.2.1-1 drum 1.1 M

Transaction Summary
=================================================================================================================================================================================================================
Install 1 Package(s)

Total download size: 1.1 M
Installed size: 3.0 M
Is this ok [y/N]: y
Downloading Packages:
couchdb-1.2.1-1.x86_64.rpm | 1.1 MB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : couchdb-1.2.1-1.x86_64 1/1
Verifying : couchdb-1.2.1-1.x86_64 1/1

Installed:
couchdb.x86_64 0:1.2.1-1

Complete!
`

Simples! Next step is to start it, confirm it’s listening on a port and test it works!

<br /> [root@hmon etc]# /etc/init.d/couchdb start<br /> Starting database server couchdb<br /> [root@hmon etc]# ps aux | grep couch<br /> root 9790 0.0 0.0 106188 1532 pts/1 S 14:41 0:00 /bin/sh -e /opt/netdev/erlang/bin/couchdb -a /etc/couchdb/default.ini -a /etc/couchdb/local.ini -b -r 0 -p /var/run/couchdb/couchdb.pid -o couchdb.stdout -e couchdb.stderr -R<br /> root 9800 0.0 0.0 106188 760 pts/1 S 14:41 0:00 /bin/sh -e /opt/netdev/erlang/bin/couchdb -a /etc/couchdb/default.ini -a /etc/couchdb/local.ini -b -r 0 -p /var/run/couchdb/couchdb.pid -o couchdb.stdout -e couchdb.stderr -R<br /> root 9801 0.7 0.1 666732 18576 pts/1 Sl 14:41 0:00 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -Bd -K true -A 4 -- -root /usr/lib64/erlang -progname erl -- -home /root -- -noshell -noinput -os_mon start_memsup false start_cpu_sup false disk_space_check_interval 1 disk_almost_full_threshold 1 -sasl errlog_type error -couch_ini /etc/couchdb/default.ini /etc/couchdb/local.ini /etc/couchdb/default.ini /etc/couchdb/local.ini -s couch -pidfile /var/run/couchdb/couchdb.pid -heart<br /> root 9834 0.0 0.0 103236 872 pts/1 S+ 14:42 0:00 grep couch<br /> root 26078 0.0 0.0 173292 1720 ? S May07 0:00 sudo -u drum setsid node /opt/netdev/drum-collab-provisioning/server/server-couch.js<br /> drum 26079 0.0 0.4 999360 70064 ? Ssl May07 18:09 node /opt/netdev/drum-collab-provisioning/server/server-couch.js<br /> [root@hmon etc]# netstat -lpn | grep 9801<br /> tcp 0 0 127.0.0.1:5984 0.0.0.0:* LISTEN 9801/beam.smp<br />

At the moment in it’s default configuration it’s only listening on 127.0.0.1 so we want to fix that!

We also want to ensure that secure_rewrites is disabled else NPM will spit out loads of errors and not work!

In /etc/couchdb/local.ini we can override the default configuration, so we want to change

<br /> [httpd]<br /> ;port = 5984<br /> ;bind_address = 127.0.0.1<br />

to

<br /> [httpd]<br /> ;port = 5984<br /> bind_address = 0.0.0.0<br /> secure_rewrites = false<br />

and then restart couchdb with /etc/init.d/couchdb restart! Now we can see couchdb listening on all ports and test it with curl

<br /> [root@hmon couchdb]# netstat -lpn | grep 5984<br /> tcp 0 0 0.0.0.0:5984 0.0.0.0:* LISTEN 10011/beam.smp<br /> [root@hmon couchdb]# curl https://localhost:5984<br /> {"couchdb":"Welcome","version":"1.2.1"}<br />

Yay, we’re alive!!!

Setting Up CouchDB Replication

Now we need to tell couchdb that it needs to replicate from the NPM master in a continuous fashion, so as the NPM master updates, so does our couchdb instance!

<br /> [root@hmon couchdb]# curl -X POST https://127.0.0.1:5984/_replicate -d '{"source":"https://isaacs.iriscouch.com/registry/", "target":"registry", "continuous":true, "create_target":true}' -H "Content-Type: application/json"<br /> {"ok":true,"_local_id":"d0818878c462afa6791440ab08348394+continuous+create_target"}<br />

And we’re off! You can interrogate how the replcation is doing by visiting the server with a webbrowser at _https://hostname:5984/utils/ it should look a little like this:

couchdb-npm-mirror

Eventually it will stop growing, I promise 😉 As of writing it’s just shy of 50GB

NPM Repository Mirror

Configuring NPM To Use Your Mirror

First we need to install some random npmjs stuff in to our couch database

<br /> git clone git://github.com/isaacs/npmjs.org.git<br /> cd npmjs.org<br /> sudo npm install -g couchapp<br /> npm install couchapp<br /> npm install semver<br /> couchapp push registry/app.js https://localhost:5984/registry<br /> couchapp push www/app.js https://localhost:5984/registry<br />

Which looks a little bit like this

<br /> [root@hmon ~]# git clone git://github.com/isaacs/npmjs.org.git<br /> Initialized empty Git repository in /root/npmjs.org/.git/<br /> remote: Counting objects: 1291, done.<br /> remote: Compressing objects: 100% (742/742), done.<br /> remote: Total 1291 (delta 609), reused 1200 (delta 531)<br /> Receiving objects: 100% (1291/1291), 649.70 KiB | 353 KiB/s, done.<br /> Resolving deltas: 100% (609/609), done.<br /> [root@hmon ~]# cd npmjs.org<br /> [root@hmon npmjs.org]# npm install -g couchapp<br /> npm http GET https://registry.npmjs.org/couchapp<br /> npm http 304 https://registry.npmjs.org/couchapp<br /> npm http GET https://registry.npmjs.org/watch<br /> npm http GET https://registry.npmjs.org/request<br /> npm http 304 https://registry.npmjs.org/request<br /> npm http 304 https://registry.npmjs.org/watch<br /> npm http GET https://registry.npmjs.org/qs<br /> npm http GET https://registry.npmjs.org/json-stringify-safe<br /> npm http GET https://registry.npmjs.org/forever-agent<br /> npm http GET https://registry.npmjs.org/tunnel-agent<br /> npm http GET https://registry.npmjs.org/http-signature<br /> npm http GET https://registry.npmjs.org/hawk<br /> npm http GET https://registry.npmjs.org/aws-sign<br /> npm http GET https://registry.npmjs.org/oauth-sign<br /> npm http GET https://registry.npmjs.org/cookie-jar<br /> npm http GET https://registry.npmjs.org/node-uuid<br /> npm http GET https://registry.npmjs.org/mime<br /> npm http GET https://registry.npmjs.org/form-data/0.0.8<br /> npm http 200 https://registry.npmjs.org/json-stringify-safe<br /> npm http GET https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-4.0.0.tgz<br /> npm http 200 https://registry.npmjs.org/forever-agent<br /> npm http GET https://registry.npmjs.org/forever-agent/-/forever-agent-0.5.0.tgz<br /> npm http 200 https://registry.npmjs.org/http-signature<br /> npm http 200 https://registry.npmjs.org/tunnel-agent<br /> npm http GET https://registry.npmjs.org/http-signature/-/http-signature-0.9.11.tgz<br /> npm http GET https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/qs<br /> npm http GET https://registry.npmjs.org/qs/-/qs-0.6.5.tgz<br /> npm http 200 https://registry.npmjs.org/aws-sign<br /> npm http GET https://registry.npmjs.org/aws-sign/-/aws-sign-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/oauth-sign<br /> npm http GET https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/cookie-jar<br /> npm http GET https://registry.npmjs.org/cookie-jar/-/cookie-jar-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/node-uuid<br /> npm http GET https://registry.npmjs.org/node-uuid/-/node-uuid-1.4.0.tgz<br /> npm http 200 https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-4.0.0.tgz<br /> npm http 200 https://registry.npmjs.org/forever-agent/-/forever-agent-0.5.0.tgz<br /> npm http 200 https://registry.npmjs.org/mime<br /> npm http GET https://registry.npmjs.org/mime/-/mime-1.2.9.tgz<br /> npm http 200 https://registry.npmjs.org/form-data/0.0.8<br /> npm http GET https://registry.npmjs.org/form-data/-/form-data-0.0.8.tgz<br /> npm http 200 https://registry.npmjs.org/http-signature/-/http-signature-0.9.11.tgz<br /> npm http 200 https://registry.npmjs.org/qs/-/qs-0.6.5.tgz<br /> npm http 200 https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/aws-sign/-/aws-sign-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/cookie-jar/-/cookie-jar-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.3.0.tgz<br /> npm http 200 https://registry.npmjs.org/node-uuid/-/node-uuid-1.4.0.tgz<br /> npm http 200 https://registry.npmjs.org/mime/-/mime-1.2.9.tgz<br /> npm http 200 https://registry.npmjs.org/form-data/-/form-data-0.0.8.tgz<br /> npm http 200 https://registry.npmjs.org/hawk<br /> npm http GET https://registry.npmjs.org/hawk/-/hawk-0.13.1.tgz<br /> npm http 200 https://registry.npmjs.org/hawk/-/hawk-0.13.1.tgz<br /> npm http GET https://registry.npmjs.org/assert-plus/0.1.2<br /> npm http GET https://registry.npmjs.org/asn1/0.1.11<br /> npm http GET https://registry.npmjs.org/ctype/0.5.2<br /> npm http GET https://registry.npmjs.org/hoek<br /> npm http GET https://registry.npmjs.org/boom<br /> npm http GET https://registry.npmjs.org/cryptiles<br /> npm http GET https://registry.npmjs.org/sntp<br /> npm http GET https://registry.npmjs.org/combined-stream<br /> npm http GET https://registry.npmjs.org/async<br /> npm http 200 https://registry.npmjs.org/ctype/0.5.2<br /> npm http 200 https://registry.npmjs.org/assert-plus/0.1.2<br /> npm http 200 https://registry.npmjs.org/asn1/0.1.11<br /> npm http 200 https://registry.npmjs.org/boom<br /> npm http GET https://registry.npmjs.org/ctype/-/ctype-0.5.2.tgz<br /> npm http GET https://registry.npmjs.org/boom/-/boom-0.4.2.tgz<br /> npm http GET https://registry.npmjs.org/assert-plus/-/assert-plus-0.1.2.tgz<br /> npm http GET https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz<br /> npm http 200 https://registry.npmjs.org/cryptiles<br /> npm http 200 https://registry.npmjs.org/combined-stream<br /> npm http GET https://registry.npmjs.org/cryptiles/-/cryptiles-0.2.1.tgz<br /> npm http GET https://registry.npmjs.org/combined-stream/-/combined-stream-0.0.4.tgz<br /> npm http 200 https://registry.npmjs.org/sntp<br /> npm http 200 https://registry.npmjs.org/ctype/-/ctype-0.5.2.tgz<br /> npm http 200 https://registry.npmjs.org/boom/-/boom-0.4.2.tgz<br /> npm http GET https://registry.npmjs.org/sntp/-/sntp-0.2.4.tgz<br /> npm http 200 https://registry.npmjs.org/assert-plus/-/assert-plus-0.1.2.tgz<br /> npm http 200 https://registry.npmjs.org/asn1/-/asn1-0.1.11.tgz<br /> npm http 200 https://registry.npmjs.org/cryptiles/-/cryptiles-0.2.1.tgz<br /> npm http 200 https://registry.npmjs.org/combined-stream/-/combined-stream-0.0.4.tgz<br /> npm http 200 https://registry.npmjs.org/sntp/-/sntp-0.2.4.tgz<br /> npm http 200 https://registry.npmjs.org/hoek<br /> npm http GET https://registry.npmjs.org/hoek/-/hoek-0.8.5.tgz<br /> npm http 200 https://registry.npmjs.org/async<br /> npm http 200 https://registry.npmjs.org/hoek/-/hoek-0.8.5.tgz<br /> npm http GET https://registry.npmjs.org/async/-/async-0.2.9.tgz<br /> npm http GET https://registry.npmjs.org/hoek<br /> npm http 200 https://registry.npmjs.org/async/-/async-0.2.9.tgz<br /> npm http GET https://registry.npmjs.org/delayed-stream/0.0.5<br /> npm http 304 https://registry.npmjs.org/hoek<br /> npm http GET https://registry.npmjs.org/hoek/-/hoek-0.9.1.tgz<br /> npm http 200 https://registry.npmjs.org/delayed-stream/0.0.5<br /> npm http GET https://registry.npmjs.org/delayed-stream/-/delayed-stream-0.0.5.tgz<br /> npm http 200 https://registry.npmjs.org/hoek/-/hoek-0.9.1.tgz<br /> npm http 200 https://registry.npmjs.org/delayed-stream/-/delayed-stream-0.0.5.tgz<br /> /root/.nvm/v0.8.15/bin/couchapp -> /root/.nvm/v0.8.15/lib/node_modules/couchapp/bin.js<br /> couchapp@0.9.1 /root/.nvm/v0.8.15/lib/node_modules/couchapp<br /> ├── watch@0.7.0<br /> └── request@2.21.0 (json-stringify-safe@4.0.0, forever-agent@0.5.0, aws-sign@0.3.0, qs@0.6.5, tunnel-agent@0.3.0, oauth-sign@0.3.0, cookie-jar@0.3.0, node-uuid@1.4.0, mime@1.2.9, http-signature@0.9.11, hawk@0.13.1, form-data@0.0.8)<br /> [root@hmon npmjs.org]# npm install couchapp<br /> npm http GET https://registry.npmjs.org/couchapp<br /> npm http 304 https://registry.npmjs.org/couchapp<br /> npm http GET https://registry.npmjs.org/watch<br /> npm http GET https://registry.npmjs.org/request<br /> npm http 304 https://registry.npmjs.org/request<br /> npm http 304 https://registry.npmjs.org/watch<br /> npm http GET https://registry.npmjs.org/qs<br /> npm http GET https://registry.npmjs.org/json-stringify-safe<br /> npm http GET https://registry.npmjs.org/forever-agent<br /> npm http GET https://registry.npmjs.org/tunnel-agent<br /> npm http GET https://registry.npmjs.org/http-signature<br /> npm http GET https://registry.npmjs.org/hawk<br /> npm http GET https://registry.npmjs.org/aws-sign<br /> npm http GET https://registry.npmjs.org/oauth-sign<br /> npm http GET https://registry.npmjs.org/cookie-jar<br /> npm http GET https://registry.npmjs.org/node-uuid<br /> npm http GET https://registry.npmjs.org/mime<br /> npm http GET https://registry.npmjs.org/form-data/0.0.8<br /> npm http 304 https://registry.npmjs.org/json-stringify-safe<br /> npm http 304 https://registry.npmjs.org/qs<br /> npm http 304 https://registry.npmjs.org/forever-agent<br /> npm http 304 https://registry.npmjs.org/tunnel-agent<br /> npm http 304 https://registry.npmjs.org/http-signature<br /> npm http 304 https://registry.npmjs.org/hawk<br /> npm http 304 https://registry.npmjs.org/aws-sign<br /> npm http 304 https://registry.npmjs.org/oauth-sign<br /> npm http 304 https://registry.npmjs.org/cookie-jar<br /> npm http 304 https://registry.npmjs.org/node-uuid<br /> npm http 304 https://registry.npmjs.org/mime<br /> npm http 304 https://registry.npmjs.org/form-data/0.0.8<br /> npm http GET https://registry.npmjs.org/assert-plus/0.1.2<br /> npm http GET https://registry.npmjs.org/asn1/0.1.11<br /> npm http GET https://registry.npmjs.org/ctype/0.5.2<br /> npm http GET https://registry.npmjs.org/boom<br /> npm http GET https://registry.npmjs.org/cryptiles<br /> npm http GET https://registry.npmjs.org/hoek<br /> npm http GET https://registry.npmjs.org/sntp<br /> npm http GET https://registry.npmjs.org/combined-stream<br /> npm http GET https://registry.npmjs.org/async<br /> npm http 304 https://registry.npmjs.org/ctype/0.5.2<br /> npm http 304 https://registry.npmjs.org/asn1/0.1.11<br /> npm http 304 https://registry.npmjs.org/assert-plus/0.1.2<br /> npm http 304 https://registry.npmjs.org/boom<br /> npm http 304 https://registry.npmjs.org/cryptiles<br /> npm http 304 https://registry.npmjs.org/hoek<br /> npm http 304 https://registry.npmjs.org/sntp<br /> npm http 304 https://registry.npmjs.org/combined-stream<br /> npm http 304 https://registry.npmjs.org/async<br /> npm http GET https://registry.npmjs.org/hoek<br /> npm http GET https://registry.npmjs.org/delayed-stream/0.0.5<br /> npm http 304 https://registry.npmjs.org/hoek<br /> npm http 304 https://registry.npmjs.org/delayed-stream/0.0.5<br /> couchapp@0.9.1 node_modules/couchapp<br /> ├── watch@0.7.0<br /> └── request@2.21.0 (json-stringify-safe@4.0.0, forever-agent@0.5.0, aws-sign@0.3.0, qs@0.6.5, tunnel-agent@0.3.0, oauth-sign@0.3.0, cookie-jar@0.3.0, node-uuid@1.4.0, mime@1.2.9, http-signature@0.9.11, hawk@0.13.1, form-data@0.0.8)<br /> [root@hmon npmjs.org]# npm install semver<br /> npm http GET https://registry.npmjs.org/semver<br /> npm http 200 https://registry.npmjs.org/semver<br /> npm http GET https://registry.npmjs.org/semver/-/semver-1.0.14.tgz<br /> npm http 200 https://registry.npmjs.org/semver/-/semver-1.0.14.tgz<br /> semver@1.0.14 node_modules/semver<br /> [root@hmon npmjs.org]# couchapp push registry/app.js https://localhost:5984/registry<br /> Preparing.<br /> Serializing.<br /> PUT https://localhost:5984/registry/_design/scratch<br /> Finished push. 1-8b4b7cde241179296b34d437d6fcbec3<br /> [root@hmon npmjs.org]# couchapp push www/app.js https://localhost:5984/registry<br /> Preparing.<br /> Serializing.<br /> PUT https://localhost:5984/registry/_design/ui<br /> Finished push. 1-7d950267677c7a20ec944e4c8385af2f<br />

Ok, now we should in theory have a completly working mirror repository of npm now!

Testing Your NPM Repository Mirror

Testing the npm repo mirror is easy enough, first we’ll make a request to the official repo then compare it to our server!

<br /> [root@hmon npmjs.org]# npm search cakes<br /> NAME DESCRIPTION AUTHOR DATE KEYWORDS<br /> mocha-cakes bdd stories add-on for mocha test framework with cucumber given/then/when syntax. =quangv 2012-06-03 15:30 mocha bdd stories cucumber test testing gherkin acceptance customer functional end-user<br /> npm http GET https://registry.npmjs.org/-/all/since?stale=update_after&startkey=1371737099041<br /> npm http 200 https://registry.npmjs.org/-/all/since?stale=update_after&startkey=1371737099041<br />

Ok the official mirror has something called mocha-cakes in it, sounds delicious!

<br /> [root@hmon couchdb]# npm search cakes<br /> npm http GET https://localhost:5984/registry/_design/app/_rewrite/-/all/since?stale=update_after&startkey=1371802019107<br /> npm http 200 https://localhost:5984/registry/_design/app/_rewrite/-/all/since?stale=update_after&startkey=1371802019107<br /> NAME DESCRIPTION AUTHOR DATE KEYWORDS<br /> mocha-cakes bdd stories add-on for mocha test framework with cucumber given/then/when syntax. =quangv 2012-06-03 15:30 mocha bdd stories cucumber test testing gherkin acceptance customer functional end-user<br />

And so does ours!!! It’s still syncing so I’m going to forgive that 404

Telling NPM To Use Our NPM Repository Mirror

We can either configure ~/.npmrc with

<br /> registry = https://localhost:5984/registry/_design/app/_rewrite<br />

or set it on the command line with

<br /> npm config set registry https://localhost:5984/registry/_design/app/_rewrite<br />

Bamo, a dozen commands later and now you too have your own working npm repository 😀