Rus - Page 21 of 25 - Opinions on building highly available software businesses

Slow SSH and SCP connections on Ubuntu

My home Ubuntu Jaunty installation often takes a good 40-60 seconds to connect to using SSH, none of the other servers I maintain have this same problem, they’re pretty much instantaneous, and today, on a Friday evening at 1am this irked me enough to fix it!

The first step is of course to make the SSH connection with debug output on.

Chill:~ idimmu$ ssh -v cordy OpenSSH_5.2p1, OpenSSL 0.9.7l 28 Sep 2006 debug1: Reading configuration data /Users/idimmu/.ssh/config debug1: Reading configuration data /etc/ssh_config debug1: Connecting to cordy [192.168.0.10] port 22. debug1: Connection established. debug1: identity file /Users/idimmu/.ssh/identity type -1 debug1: identity file /Users/idimmu/.ssh/id_rsa type -1 debug1: identity file /Users/idimmu/.ssh/id_dsa type 2 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.1p1 Debian-5ubuntu1 debug1: match: OpenSSH_5.1p1 Debian-5ubuntu1 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.2 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024 debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: Host 'cordy' is known and matches the RSA host key. debug1: Found key in /Users/idimmu/.ssh/known_hosts:7 debug1: ssh_rsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received

< 30 – 40 second pause occurs here >

debug1: Authentications that can continue: publickey,password debug1: Next authentication method: publickey debug1: Trying private key: /Users/idimmu/.ssh/identity debug1: Trying private key: /Users/idimmu/.ssh/id_rsa debug1: Offering public key: /Users/idimmu/.ssh/id_dsa debug1: Server accepts key: pkalg ssh-dss blen 433 debug1: read PEM private key done: type DSA debug1: Authentication succeeded (publickey). debug1: channel 0: new [client-session] debug1: Requesting no-more-sessions@openssh.com debug1: Entering interactive session. debug1: Requesting authentication agent forwarding. Linux Cordy 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC 2009 i686 Last login: Sat Jan 23 01:22:08 2010 from chill.local idimmu@Cordy:~$

A quick Google for SSH2_MSG_SERVICE_ACCEPT and a read of man sshd_config gave me:

UseDNS Specifies whether sshd(8) should look up the remote host name and check that the resolved host name for the remote IP address maps back to the very same IP address. The default is ``yes''.

So I simply added

UseDNS no

to the end of /etc/ssh/sshd_config and issued an sudo /etc/init.d/ssh reload and bamo, instant SSH access and I can sleep peacefully!

Proper use of grails:exec

by rus

Whilst trying to run mvn grails:exec with more complicated requests it kept erroring:

Chill:superted idimmu$ mvn grails:exec -Dcommand="jack-be-nimble net.idimmu User Profile" [INFO] Scanning for projects... [INFO] Searching repository for plugin with prefix: 'grails'. [INFO] ------------------------------------------------------------------------ [INFO] Building Unnamed - net.idimmu:gymlife:war:1.0-SNAPSHOT [INFO] task-segment: [grails:exec] (aggregator-style) [INFO] ------------------------------------------------------------------------ [INFO] [grails:exec] [INFO] Using Grails 1.2.0 Running pre-compiled script [INFO] ------------------------------------------------------------------------ [ERROR] BUILD ERROR [INFO] ------------------------------------------------------------------------ [INFO] Unable to start Grails
Embedded error: java.lang.reflect.InvocationTargetException
[INFO] ————————————————————————
[INFO] For more information, run Maven with the -e switch
[INFO] ————————————————————————
[INFO] Total time: 14 seconds
[INFO] Finished at: Thu Jan 07 22:49:41 GMT 2010
[INFO] Final Memory: 32M/81M
[INFO] ————————————————————————

After some playing about, I finally found the correct runes

Chill:superted idimmu$ mvn grails:exec -Dcommand="jack-be-nimble" -Dargs="net.idimmu User Profile" [INFO] Scanning for projects... [INFO] Searching repository for plugin with prefix: 'grails'. [INFO] ------------------------------------------------------------------------ [INFO] Building Unnamed - net.idimmu:superted:war:1.0-SNAPSHOT [INFO] task-segment: [grails:exec] (aggregator-style) [INFO] ------------------------------------------------------------------------ [INFO] [grails:exec] [INFO] Using Grails 1.2.0

Running script /Users/idimmu/Documents/stsworkspace/superted/plugins/nimble-0.3-SNAPSHOT/scripts/JackBeNimble.groovy Environment set to development Using package net.idimmu to create custom classes Setting up nimble with custom User domain class: User Setting up nimble with custom Profile domain class: Profile [echo] Jack be nimble [echo] Jack be quick [echo] Jack jump over [echo] The candlestick. [copy] Copying 1 file to /Users/idimmu/Documents/stsworkspace/superted/grails-app/conf [copy] Copying 1 file to /Users/idimmu/Documents/stsworkspace/superted/grails-app/conf [copy] Copying 1 file to /Users/idimmu/Documents/stsworkspace/superted/grails-app/conf [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/superted/grails-app/domain/net/idimmu [copy] Copying 15 files to /Users/idimmu/Documents/stsworkspace/superted/grails-app/views/templates/nimble [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/superted/src/sass [copy] Copying 1 file to /Users/idimmu/Documents/stsworkspace/superted/src/sass [copy] Copying 1 file to /Users/idimmu/Documents/stsworkspace/superted/src/sass [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESSFUL [INFO] ------------------------------------------------------------------------ [INFO] Total time: 16 seconds [INFO] Finished at: Thu Jan 07 22:56:52 GMT 2010 [INFO] Final Memory: 41M/81M [INFO] ------------------------------------------------------------------------

Woo hoo!

Latest release information is not available for plugin ‘nimble’, specify concrete release to install

by rus

Whilst trying to install the Nimble plugin for Grails using Maven I ran in to a spot of bother!

Chill:superted idimmu$ mvn grails:install-plugin -DpluginName=nimble [INFO] Scanning for projects... [INFO] Searching repository for plugin with prefix: 'grails'. [INFO] ------------------------------------------------------------------------ [INFO] Building Unnamed - idimmu.net:superted:war:1.0-SNAPSHOT [INFO] task-segment: [grails:install-plugin] (aggregator-style) [INFO] ------------------------------------------------------------------------ [INFO] [grails:install-plugin] [INFO] Using Grails 1.2.0 Running pre-compiled script Environment set to development Reading remote plugin list ... Reading remote plugin list ... Latest release information is not available for plugin 'nimble', specify concrete release to install

An investigation in to the available plugins yielded

Chill:superted idimmu$ mvn grails:list-plugins [INFO] Scanning for projects... [INFO] Searching repository for plugin with prefix: 'grails'.



nimble              <0.3-SNAPSHOT (?)>	 --  Nimble

So a bit of reading and thinking and specifically talking to another Grails developer, I found the solution

Chill:superted idimmu$ mvn grails:install-plugin -DpluginName=nimble -DpluginVersion=0.3-SNAPSHOT [INFO] Scanning for projects... [INFO] Searching repository for plugin with prefix: 'grails'. [INFO] ------------------------------------------------------------------------ [INFO] Building Unnamed - idimmu.net:superted:war:1.0-SNAPSHOT [INFO] task-segment: [grails:install-plugin] (aggregator-style) [INFO] ------------------------------------------------------------------------ [INFO] [grails:install-plugin] [INFO] Using Grails 1.2.0 Running pre-compiled script Environment set to development Reading remote plugin list ... Reading remote plugin list ... [get] Getting: https://plugins.grails.org/grails-nimble/tags/RELEASE_0_3-SNAPSHOT/grails-nimble-0.3-SNAPSHOT.zip [get] To: /Users/idimmu/.grails/1.2.0/plugins/grails-nimble-0.3-SNAPSHOT.zip

Integrating Grails and Maven

by rus

I’ve started investigating Grails for some projects and have been making some good headway and I like to use Maven for everything as it’s amazing, and generating projects is easy.

Chill:stsworkspace idimmu$ mvn org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-4:generate -DarchetypeGroupId=org.grails -DarchetypeArtifactId=grails-maven-archetype -DarchetypeVersion=1.2.0 -DgroupId=idimmu.net -DartifactId=superted [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Building Maven Default Project [INFO] task-segment: [org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-4:generate] (aggregator-style) [INFO] ------------------------------------------------------------------------ [INFO] Preparing archetype:generate [INFO] No goals needed for project - skipping [INFO] Setting property: classpath.resource.loader.class => 'org.codehaus.plexus.velocity.ContextClassLoaderResourceLoader'. [INFO] Setting property: velocimacro.messages.on => 'false'. [INFO] Setting property: resource.loader => 'classpath'. [INFO] Setting property: resource.manager.logwhenfound => 'false'. [INFO] [archetype:generate] [INFO] Generating project in Interactive mode [WARNING] No archetype repository found. Falling back to central repository (https://repo1.maven.org/maven2). [WARNING] Use -DarchetypeRepository= if archetype's repository is elsewhere. Define value for version: 1.0-SNAPSHOT: : Confirm properties configuration: groupId: idimmu.net artifactId: superted version: 1.0-SNAPSHOT package: idimmu.net Y: : [INFO] ---------------------------------------------------------------------------- [INFO] Using following parameters for creating OldArchetype: grails-maven-archetype:1.2.0 [INFO] ---------------------------------------------------------------------------- [INFO] Parameter: groupId, Value: idimmu.net [INFO] Parameter: packageName, Value: idimmu.net [INFO] Parameter: package, Value: idimmu.net [INFO] Parameter: artifactId, Value: superted [INFO] Parameter: basedir, Value: /Users/idimmu/Documents/stsworkspace [INFO] Parameter: version, Value: 1.0-SNAPSHOT [WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference : template = archetype-resources/pom.xml [line 147,column 20] : ${java.version} is not a valid reference. [WARNING] org.apache.velocity.runtime.exception.ReferenceException: reference : template = archetype-resources/pom.xml [line 149,column 23] : ${java.home} is not a valid reference. [INFO] ********************* End of debug info from resources from generated POM *********************** [INFO] OldArchetype created in dir: /Users/idimmu/Documents/stsworkspace/superted [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESSFUL [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15 seconds [INFO] Finished at: Thu Jan 07 21:45:39 GMT 2010 [INFO] Final Memory: 12M/79M [INFO] ------------------------------------------------------------------------

Then of course we initialise the project

Chill:superted idimmu$ mvn initialize [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Building Unnamed - idimmu.net:gymlife:war:1.0-SNAPSHOT [INFO] task-segment: [initialize] [INFO] ------------------------------------------------------------------------ [INFO] [grails:validate {execution: default}] [INFO] No Grails application found - skipping validation. [INFO] [grails:init {execution: default}] [INFO] Cannot read application info, so initialising new application. [INFO] Using Grails 1.2.0 Running pre-compiled script Environment set to development Warning, target causing name overwriting of name default [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/src/java [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/src/groovy [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/controllers [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/services [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/domain [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/taglib [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/utils [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/views [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/views/layouts [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/i18n [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/conf [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/test [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/test/unit [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/test/integration [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/scripts [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/web-app [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/web-app/js [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/web-app/css [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/web-app/images [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/web-app/META-INF [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/lib [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/conf/spring [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app/conf/hibernate [propertyfile] Creating new property file: /Users/idimmu/Documents/stsworkspace/gymlife/application.properties [copy] Copying 1 resource to /Users/idimmu/Documents/stsworkspace/gymlife [unjar] Expanding: /Users/idimmu/Documents/stsworkspace/gymlife/grails-shared-files.jar into /Users/idimmu/Documents/stsworkspace/gymlife [delete] Deleting: /Users/idimmu/Documents/stsworkspace/gymlife/grails-shared-files.jar [copy] Copying 1 resource to /Users/idimmu/Documents/stsworkspace/gymlife [unjar] Expanding: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app-files.jar into /Users/idimmu/Documents/stsworkspace/gymlife [delete] Deleting: /Users/idimmu/Documents/stsworkspace/gymlife/grails-app-files.jar [copy] Copying 1 resource to /Users/idimmu/Documents/stsworkspace/gymlife/target/integration-files [unjar] Expanding: /Users/idimmu/Documents/stsworkspace/gymlife/target/integration-files/grails-integration-files.jar into /Users/idimmu/Documents/stsworkspace/gymlife/target/integration-files [delete] Deleting: /Users/idimmu/Documents/stsworkspace/gymlife/target/integration-files/grails-integration-files.jar [copy] Copying 3 files to /Users/idimmu/Documents/stsworkspace/gymlife Created Eclipse project files. Installing plugin hibernate-1.2.0 [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/plugins/hibernate-1.2.0 [unzip] Expanding: /Users/idimmu/.grails/1.2.0/plugins/grails-hibernate-1.2.0.zip into /Users/idimmu/Documents/stsworkspace/gymlife/plugins/hibernate-1.2.0 Resolving plugin JAR dependencies Executing hibernate-1.2.0 plugin post-install script ... Plugin hibernate-1.2.0 installed Installing plugin tomcat-1.2.0 [mkdir] Created dir: /Users/idimmu/Documents/stsworkspace/gymlife/plugins/tomcat-1.2.0 [unzip] Expanding: /Users/idimmu/.grails/1.2.0/plugins/grails-tomcat-1.2.0.zip into /Users/idimmu/Documents/stsworkspace/gymlife/plugins/tomcat-1.2.0 Executing tomcat-1.2.0 plugin post-install script ... Plugin tomcat-1.2.0 installed Plug-in provides the following new scripts: ------------------------------------------ grails tomcat Found events script in plugin tomcat Created Grails Application at /Users/idimmu/Documents/stsworkspace/superted [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESSFUL [INFO] ------------------------------------------------------------------------ [INFO] Total time: 16 seconds [INFO] Finished at: Thu Jan 07 21:46:24 GMT 2010 [INFO] Final Memory: 48M/81M [INFO] ------------------------------------------------------------------------

Technorati Have Crappy Systems Designers

by rus

I’ve signed up to Technorati and to prove I own this blog they want me to post an entry with this code in it:

4VPEAZA86VHN

or maybe this code

7W3G7P94H3PU

They don’t seem to be sure and the email they sent me and their website both say something different ..

Most sites that are better, such as Google or even MS Live/Bing/Bling/whatever they’re calling it this week ask you to create a file on the server, but noooooooo Technorati are forcing me to actually make a post advertising their service .. I really hope it has a happy ending!!! 😉

Python support for SSL and HTTPS is not installed

by rus

I wanted to give Google’s new Go language a try which requires checking out the source code with Mercurial.

Mercurial was installed from ports on my Mac with

sudo port install mecurial

But checking out the code threw an error.

Chill:~ rus$ hg clone -r release https://go.googlecode.com/hg/ $GOROOT abort: Python support for SSL and HTTPS is not installed Exception exceptions.AttributeError: "'httpsrepository' object has no attribute 'urlopener'" in > ignored

After some hunting I found the answer is to install the python socket ssl library from ports

Chill:~ rus$ sudo port install py25-socket-ssl ---> Fetching py25-socket-ssl ---> Verifying checksum(s) for py25-socket-ssl ---> Extracting py25-socket-ssl ---> Configuring py25-socket-ssl ---> Building py25-socket-ssl ---> Staging py25-socket-ssl into destroot ---> Installing py25-socket-ssl @2.5.4_0 ---> Activating py25-socket-ssl @2.5.4_0 ---> Cleaning py25-socket-ssl

Which then let me check out the source for Go!

Chill:~ rus$ hg clone -r release https://go.googlecode.com/hg/ $GOROOT requesting all changes adding changesets adding manifests adding file changes added 3976 changesets with 16799 changes to 2931 files updating working directory 1640 files updated, 0 files merged, 0 files removed, 0 files unresolved

Now it’s time to compile 😀

Error: Could not open command file ‘/var/log/nagios/rw/nagios.cmd’ for update! on RedHat

by rus

I’m currently setting up some new clusters under RedHat and each cluster is getting it’s own Nagios instance, trying to use the web based management interface however threw and error.

I’ve seen this error before on Ubuntu and was getting it again under RedHat. Of course I revisited my Ubuntu solution and realised that it didn’t help at all, due to using dpkg overrides, also the situation was very different!

root@nagios:/var/log/nagios/rw# ls -al total 8 drwxrwxr-x 2 nagios apache 4096 Oct 30 13:37 . drwxrwxr-x 5 nagios apache 4096 Oct 30 13:40 ..

The file didn’t exist 🙁

A quick scan of a working system showed :

prw-rw---- 1 nagios nagcmd 0 2009-10-15 13:32 nagios.cmd

It’s a pipe! Woohoo.

So we need to create it

root@nagios:/var/log/nagios/rw# mknod nagios.cmd p root@nagios:/var/log/nagios/rw# chown nagios:apache nagios.cmd root@nagios:/var/log/nagios/rw# chmod 660 nagios.cmd root@nagios:/var/log/nagios/rw# ls -la total 8 drwxrwxr-x 2 nagios apache 4096 Oct 30 13:37 . drwxrwxr-x 5 nagios apache 4096 Oct 30 13:43 .. prw-rw---- 1 nagios apache 0 Oct 30 13:37 nagios.cmd root@nagios:/var/log/nagios/rw# /etc/init.d/nagios restart

A quick check of the site and .. same error? Still broken? Noooooooooooo

Lets have another look ..

root@mm2su0:/var/log/nagios/rw# ls -al total 8 drwxrwxr-x 2 nagios apache 4096 Oct 30 13:45 . drwxrwxr-x 5 nagios apache 4096 Oct 30 13:45 .. prw-rw---- 1 nagios nagios 0 Oct 30 13:45 nagios.cmd

Nice, nagios changed the permissions for us so apache can’t write to it. I’m not setting apache to run as the nagios user 🙁

A look in the init file for nagios shows that it actually manages the file itself, so we didn’t need to actually make it (strange that it wasn’t there when I looked then ..) The init file actually handles creation and removal of the file :

root@nagios:/var/log/nagios/rw# /etc/init.d/nagios stop Stopping nagios: done. root@nagios:/var/log/nagios/rw# ls root@nagios:/var/log/nagios/rw# /etc/init.d/nagios start Starting nagios: done. root@nagios:/var/log/nagios/rw# ls nagios.cmd root@nagios:/var/log/nagios/rw# ls nagios.cmd root@nagios:/var/log/nagios/rw# ls -al total 8 drwxrwxr-x 2 nagios apache 4096 Oct 30 13:52 . drwxrwxr-x 5 nagios apache 4096 Oct 30 13:52 .. prw-rw---- 1 nagios nagios 0 Oct 30 13:52 nagios.cmd

To fix this I’m going to stick the apache user in the nagios group.

[/code]
root@nagios:/var/log/nagios/rw# usermod -G nagios apache
root@nagios:/var/log/nagios/rw# /etc/init.d/httpd restart
[/code]

No more error, problem solved!

If you want to become a more advanced Nagios administrator, I recommend Nagios by O’Reilly. It’s full of best practice implementation advice.

Creating A Local and HTTP RedHat Yum Repository

by rus

We’re rolling out a RedHat platform for a major product delivery to a client in the next couple of weeks, which is a great chance for me to build a fresh platform using all the skills I’ve acquired and actually do things properly (TM).

We’re using RedHat Enterprise Linux 5 because this is a very critical deployment and we want an Operating System that is certified to both our hardware and software applications, something that unfortunately Ubuntu does not deliver.

The first stage of the deployment is to create our own RedHat RPM repository on the local network, so we can easily create servers.

I’m going to cover setting up 2 types of repository, initially a file system based repo and then an http based repo so that other servers can talk to the repo and snag RPMs from it.

To start with a repository is just a path with RPMs in it, regardless of file hierarchy, so we need to create this. I’m working with the RHEL 5 DVD so my first task is to simply copy the contents of the DVD on to the server’s hard drive.

First we mount the DVD

[rus@redhat ~]$ sudo mkdir /media/cdrom [rus@redhat ~]$ sudo mount /dev/cdrom /media/cdrom/

And then we copy across the entire of the DVD to where we want the file system based repo to be!

[rus@redhat ~]$ sudo mkdir -p /var/repo/rhel5 [rus@redhat ~]$ sudo cp -r /media/cdrom/* /var/repo/rhel5 [rus@redhat ~]$ find /var/repo/rhel5 -name *.rpm | wc -l 3180

Ok, now we have a lovely directory structure with over 3000 RPMs just gagging to be installed in it 😉

Next up we need to create the Yum repository metadata, to do this we first need to install a tool called createrepo. Fortunately this is on the DVD image we just copied to our hard drive so we can sneakily install it with rpm -i 😉

[rus@redhat Server]$ cd /var/repo/rhel5/Server/ [rus@redhat Server]$ sudo rpm -i createrepo-0.4.11-3.el5.noarch.rpm warning: createrepo-0.4.11-3.el5.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 37017186

Tada, createrepo is now installed! Do not worry about the NOKEY error, that will be explained and fixed shortly!

To create the metadata we simply run createrepo . in the repo directory.

[rus@redhat Server]$ cd /var/repo/ [rus@redhat repo]$ sudo createrepo . 3180/3180 - rhel5/VT/Virtualization-it-IT-5.2-11.noarch.rpm .rpm3.x86_64.rpm Saving Primary metadata Saving file lists metadata Saving other metadata [rus@redhat repo]$ ls repodata rhel5 [rus@redhat repo]$ ls -al repodata/ total 13460 drwxr-xr-x 2 root root 4096 Oct 20 23:00 . drwxr-xr-x 4 root root 4096 Oct 20 23:00 .. -rw-r--r-- 1 root root 3178251 Oct 20 23:00 filelists.xml.gz -rw-r--r-- 1 root root 9431119 Oct 20 23:00 other.xml.gz -rw-r--r-- 1 root root 1108342 Oct 20 23:00 primary.xml.gz -rw-r--r-- 1 root root 951 Oct 20 23:00 repomd.xml

As you can see we have lovely metadata now for our 3180 RPMs!

repomd.xml this is the file that describes the other metadata files. It is like an index file to point to the other files. It contains timestamps and checksums for the other files. This lets a client download this one, small file and know if anything else has changed. This also means that cryptographically (ex: gpg) signing this one file can ensure repository integrity.

primary.xml.gz this file stores the primary metadata information. This includes information such as name, epoch, version, release, architecture, file size, file location, description, summary, format, checksums header byte-ranges, dependencies, provides, conflicts, obsoletes, suggests, recommends,file lists for the package for CERTAIN files – specifically files matching: /etc*, *bin/*, /usr/lib/sendmail

filelists.xml.gz this file stores the complete file and directory listings for the packages. The package is identified by: name, epoch, version, release, architecture and package checksum id.

other.xml.gz this file currently only stores the changelog data from packages. However, this file could be used for any other additional metadata that could be useful for clients.

Ok, we now have our file system based repo, so we now need to tell Yum all about it!

Yum repository information is located in /etc/yum.repos.d as a series of files, it is possible to define multiple repositories in a single file, but I prefer to limit each file to one repository as in my opinion this makes it easier to manage, especially when implementing tools like Puppet.

Our repository is a local rhel repo, so create it vi /etc/yum.repo.d/rhel-local.repo

[rhel-local] gpgcheck=1 name=Red Hat Linux $releasever - $basearch - DVD baseurl=file:///var/repo

What’s going on here is quite simple:

We’ve created a repo labeled rhel-local. We’ve told Yum that the RPMs are signed using GPG, we’ve given it a name, and we’ve defined the baseurl of the repo to be a local file path!

To test we simply run yum list and you’ll see that packages are now available in the rhel-local repo!

That’s great, but we have to get on to the real task of creating a HTTP based Yum repository so we can use it to start building servers!! We’re going to serve the repo via the Apache HTTPD server so the first logical thing to do is to install it, using our new fancy pants local repo 😉

rus@redhat ~]$ sudo yum install httpd Loaded plugins: rhnplugin, security This system is not registered with RHN. RHN support will be disabled. Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.2.3-31.el5 set to be updated --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd --> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd --> Running transaction check ---> Package apr.x86_64 0:1.2.7-11.el5_3.1 set to be updated ---> Package apr-util.x86_64 0:1.2.7-7.el5_3.2 set to be updated --> Processing Dependency: libpq.so.4()(64bit) for package: apr-util --> Running transaction check ---> Package postgresql-libs.x86_64 0:8.1.11-1.el5_1.1 set to be updated --> Finished Dependency Resolution
Dependencies Resolved
=========================================================================
Package Arch Version Repository Size
=========================================================================
Installing:
httpd x86_64 2.2.3-31.el5 rhel-local 1.2 M
Installing for dependencies:
apr x86_64 1.2.7-11.el5_3.1 rhel-local 119 k
apr-util x86_64 1.2.7-7.el5_3.2 rhel-local 75 k
postgresql-libs x86_64 8.1.11-1.el5_1.1 rhel-local 195 k
Transaction Summary
=========================================================================
Install 4 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Total download size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
————————————————————————-
Total 324 MB/s | 1.6 MB 00:00
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 37017186
Public key for apr-util-1.2.7-7.el5_3.2.x86_64.rpm is not installed

Oh god no, what is this error? Wait, that looks similar to the rpm NOKEY error we saw earlier! Relax, it’s simple!

Remember we set the repository to be GPG signed? Well, it is, but we don’t have the GPG key installed that we need to use to compare to the signings! This is easily fixable as the GPG keys came on the DVD, which we copied to the hard drive 😉

[rus@redhat ~]$ sudo rpm --import /var/repo/rhel5/RPM-GPG-KEY-redhat-beta /var/repo/rhel5/RPM-GPG-KEY-redhat-release

Now we can retry!

[rus@redhat ~]$ sudo yum install httpd Loaded plugins: rhnplugin, security This system is not registered with RHN. RHN support will be disabled. Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.2.3-31.el5 set to be updated --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: httpd --> Processing Dependency: libapr-1.so.0()(64bit) for package: httpd --> Running transaction check ---> Package apr.x86_64 0:1.2.7-11.el5_3.1 set to be updated ---> Package apr-util.x86_64 0:1.2.7-7.el5_3.2 set to be updated --> Processing Dependency: libpq.so.4()(64bit) for package: apr-util --> Running transaction check ---> Package postgresql-libs.x86_64 0:8.1.11-1.el5_1.1 set to be updated --> Finished Dependency Resolution
Dependencies Resolved
==============================================================================
Package Arch Version Repository Size
==============================================================================
Installing:
httpd x86_64 2.2.3-31.el5 rhel-local 1.2 M
Installing for dependencies:
apr x86_64 1.2.7-11.el5_3.1 rhel-local 119 k
apr-util x86_64 1.2.7-7.el5_3.2 rhel-local 75 k
postgresql-libs x86_64 8.1.11-1.el5_1.1 rhel-local 195 k
Transaction Summary
==============================================================================
Install 4 Package(s)
Update 0 Package(s)
Remove 0 Package(s)
Total download size: 1.6 M
Is this ok [y/N]: y
Downloading Packages:
——————————————————————————
Total 559 MB/s | 1.6 MB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : apr 1/4
Installing : postgresql-libs 2/4
Installing : apr-util 3/4
Installing : httpd 4/4
Installed:
httpd.x86_64 0:2.2.3-31.el5
Dependency Installed:
apr.x86_64 0:1.2.7-11.el5_3.1 apr-util.x86_64 0:1.2.7-7.el5_3.2
postgresql-libs.x86_64 0:8.1.11-1.el5_1.1
Complete!

Disco, all done.

RedHat has a quirk though, a default install of most services means that the service will not start if the box is rebooted, in fact after installing an RPM of a service, like Apache it doesn’t even start the daemon! So we must do both, register the service to be started each boot and then start the server up!

[rus@redhat ~]$ sudo /sbin/chkconfig httpd on [rus@redhat ~]$ sudo /etc/init.d/httpd start Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName [ OK ]

Using chkconfig we can tell the system to start the service on boot!

Now we just need to configure Apache to use the repo we’ve previously set up!

There are an infinite number of ways to configure Apache to serve data, but I’m going to do it the simple and quick way and just use an Alias to do it. Create /etc/httpd/conf.d/repo.conf

Alias /repo/ "/var/repo/"

Now we just need to reload Apache

[rus@redhat ~]$ sudo /etc/init.d/httpd reload Reloading httpd: [ OK ]

And we should be able to access the all important repomd.xml file with lynx :

[rus@redhat ~]$ lynx https://localhost/repo/repodata/repomd.xml

If Apache throws a 403 forbidden error error this could very well be SELinux getting in the way. If you are using SELinux you will need to change the Security Context of all the repo files so that Apache can read them!

[rus@redhat ~]$ sudo chcon -R -t httpd_sys_content_t /var/repo/

Lynx should now be able to download that file successfully.

The last thing to do now that we can retrieve files is to configure Yum to use the web based repo!

First we’re going to disable the existing local file server repo /etc/yum.repos.d/rhel-local.repo with the enabled=0 flag:

[rhel-local] gpgcheck=1 enabled=0 name=Red Hat Linux $releasever - $basearch - DVD baseurl=file:///var/repo

And create our new HTTP repo /etc/yum.repos.d/rhel-www.repo

[rhel-www] gpgcheck=1 name=Red Hat Linux $releasever - $basearch - DVD baseurl=https://localhost/repo/

To check it’s working we then just issue the following Yum commands to first clean up the repo and to then display their status!

[rus@redhat yum.repos.d]$ sudo yum clean all Loaded plugins: rhnplugin, security Cleaning up Everything [rus@redhat yum.repos.d]$ sudo yum repolist all Loaded plugins: rhnplugin, security This system is not registered with RHN. RHN support will be disabled. rhel-www | 951 B 00:00 rhel-www/primary | 1.1 MB 00:00 rhel-www 3180/3180 repo id repo name status rhel-debuginfo Red Hat Enterprise Linux 5Server - x86_64 - Debu disabled rhel-local Red Hat Linux 5Server - x86_64 - DVD disabled rhel-www Red Hat Linux 5Server - x86_64 - DVD enabled: 3,180 repolist: 3,180

Voila, one HTTP based RedHat Enterprise Linux RPM repository!

O’Reilly’s Fedora Linux covers everything you need to know about dealing with Redhat systems and is a recommended read for anyone who wants to seriously get in to Redhat administration.

Bash Shortcuts

by rus

Meta-< -> go to beginning of history file
Meta-> -> go to end of history file
Meta-f -> go to next word in command line
Meta-b -> go to previous word in command line
Meta-d -> delete next word in command line (from the actual position of the prompt)

Ctrl-a -> go to the start of command line
Ctrl-e -> go to the end of command line
Ctrl-p -> previous command in history
Ctrl-n -> next command in history
Ctrl-f -> next character in command line
Ctrl-b -> previous character in command line
Ctrl-r -> reverse search in history file
Ctrl-d -> delete current character
Ctrl-k -> delete from the prompt to the end of command line
Ctrl-_ -> undo (yes, but limited)

O’Reilly’s Bash Cookbook covers everything you’d ever need to know about Bash, including all the short cuts. Thoroughly recommended.

Burning an XVid to DVD in Ubuntu Jaunty from the command line

by rus

I needed to burn an XVid of one of my performances to DVD last night, remotely, as my MythBuntu box was being used by someone else to watch some crappy TV and I wanted to show off! It was a pretty simple process, just involving a bit of a wait during the transcode 🙂

First you will need to make install the relevent packages:

sudo aptitude install dvdauthor ffmpeg ~nlibav.+-unstripped.+

If you already have dvdauthor and ffmpeg installed, you must install the unstripped libav packages as they contain a tool called mpeg2video and it’s this that does the heavy grunt work!

Next we transcode the movie and create a DVD compatible MPEG. This is the part that takes the longest to do.

ffmpeg -i performance.avi -target dvd -aspect 16:9 -sameq performance.mpg

Now we create the DVD structure, the VIDEO_TS etc. etc.

mkdir DVD dvdauthor --title -f performance.mpg -o DVD dvdauthor -T -o DVD

Which results in a structure that looks a lot like this:

DVD DVD/VIDEO_TS DVD/VIDEO_TS/VTS_01_1.VOB DVD/VIDEO_TS/VIDEO_TS.BUP DVD/VIDEO_TS/VTS_01_0.IFO DVD/VIDEO_TS/VTS_01_3.VOB DVD/VIDEO_TS/VTS_01_2.VOB DVD/VIDEO_TS/VIDEO_TS.IFO DVD/VIDEO_TS/VTS_01_0.BUP DVD/AUDIO_TS

Then we create an iso image from that structure which we can then use to burn to a blank DVD

mkisofs -dvd-video -o performance.iso DVD

There are a million ways to burn the iso image, I actually SCP’d it to my MacBook and used the Disk Utility tool to burn it, but from the Ubuntu CD DVD Burning documentation you can use a tool called wodim!

You will need to install wodim

apt-get install wodim

list available DVD writers

wodim --devices wodim: Overview of accessible drives (1 found) : ------------------------------------------------------------------------- 0 dev='/dev/scd0' rwrw-- : 'LITE-ON' 'DVDRW SOHW-1633S' -------------------------------------------------------------------------

Insert a disk, then burn!

wodim dev=/dev/scd0 driveropts=burnfree -v -data performance.iso