I had a crazy weird bug today whilst running Puppet on one of our CentOS boxes where no packages were being installed. A quick investigation with yum yielded the following scenario!
Id I tried to use yum as root or with sudo it gave the following error ..
[idimmu@server ~]$ sudo yum check-update
rpmdb: Lock table is out of available locker entries
error: db4 error(22) from db->close: Invalid argument
error: cannot open Providename index using db3 - Cannot allocate memory (12)
Repository update is listed more than once in the configuration
Repository base is listed more than once in the configuration
Setting up repositories
https://www.mirrorservice.org/sites/mirror.centos.org/Null/updates/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Cannot open/read repomd.xml file for repository: update
failure: repodata/repomd.xml from update: [Errno 256] No more mirrors to try.
Error: failure: repodata/repomd.xml from update: [Errno 256] No more mirrors to try.
But worked fine if I did it as a non root user ..
[idimmu@server ~]$ yum check-update
Repository update is listed more than once in the configuration
Repository base is listed more than once in the configuration
Setting up repositories
Reading repository metadata in from local files
acl.i386 2.2.23-5.4.el4 base
apr-util.i386 0.9.4-22.el4 base
at.i386 3.1.8-82.el4 base
audit.i386 1.0.16-3.el4 base
I did a little bit of investigating with Google as to how yum works and discovered a few things..
https://www.mirrorservice.org/sites/mirror.centos.org/Null/updates/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
The Null in the URl stood out as an obvious case of “there’s something wrong here”, so looking at the yum.conf file ..
baseurl=https://www.mirrorservice.org/sites/mirror.centos.org/$releasever/os/$basearch/
You can see that the NULL is coming from the $releasever variable. So, how is that variable set?
At the top of the yum.conf there is
distroverpkg=centos-release
What yum does is check which version of the package specified by distroverpkg is installed, and assign that to $releasever.
Great, but why does yum think the version is NULL? Well, a search for the first error ..
rpmdb: Lock table is out of available locker entries
resulted in this site which explained everything!
Basically RPM uses a Berkley database and due to too many instances of rpm dieing (the server is an old server) it left locks on the database that were never cleared. In order to fix the problem we have to remove the locks, by unfortunately, deleting the database and recreating it!
Backup the database directory /var/lib/rpm first:
tar cvzf rpmdb-backup.tar.gz /var/lib/rpm
Delete the locked Berkeley databases that rpm uses:
rm /var/lib/rpm/__db.00*
Force rpm to rebuild the databases
rpm --rebuilddb
Now, check the database to make sure it has been rebuilt correctly:
rpm -qa | sort
Problem solved, my database is no longer corrupt and Puppet is installing packages again!
O’Reilly’s Fedora Linux covers everything you need to know about dealing with Redhat systems and is a recommended read for anyone who wants to seriously get in to Redhat administration.