google-apis-mavenized does not compile

by rus

I need to write some backup software for our hosted Google Apps, so as we’re a Java shop I’m going to be using the Java API. Also we’re heavy users of Hudson and Maven so want to get the checked out source building with that.

There is a Google Code project called google-apis-mavenized which looks like it has done most of the work for us, but it does not build when it’s checked out.

 

rus@builder:/proxy/rus$ svn checkout https://google-apis-mavenized.googlecode.com/svn/trunk/gdata-mavenized/ gdata-mavenized

 

rus@builder:/proxy/rus/$ cd gdata-mavenized

rus@builder:/proxy/rus/gdata-mavenized$ mvn install

 

/proxy/rus/gdata-mavenized/gdata-apis/client/src/main/java/com/google/gdata/data/extensions/ExtendedProperty.java:[20,31] cannot find symbol

symbol : class CoreErrorDomain

location: package com.google.gdata.client

 

After some investigation the gdata-mavenized project symlinks in files from the original gData project, but presumably due to the API constantly changing, the maintainers of google-apis-mavenized got bored or fed up so didn’t symlink in newer files.

A really quick way to fix this is simply to delete the main google directory that contains all the symlinks, and just symlink that back to the real directory that contains all the source files!

 

rus@builder:/proxy/rus/gdata-mavenized/gdata-apis/client/src/main/java/com$ rm -rf google/

rus@builder:/proxy/rus/gdata-mavenized/gdata-apis/client/src/main/java/com$ ln -s ../../../../../../gdata-external/java/src/com/google/ .

 

Bamo, problem solved!

 

rus@builder:/proxy/rus/gdata-mavenized$ mvn install

[INFO] Scanning for projects…

[INFO] Reactor build order:

[INFO] google-gdata-mavenized-1-SNAPSHOT

[INFO] google-gdata-apis:1-SNAPSHOT

[INFO] gdata-client:1-SNAPSHOT

[INFO] gdata-appsforyourdomain:1-SNAPSHOT

[INFO] gdata-base:1-SNAPSHOT

[INFO] gdata-calendar:1-SNAPSHOT

[INFO] gdata-codesearch-1-SNAPSHOT

[INFO] gdata-photos:1-SNAPSHOT

[INFO] gdata-spreadsheet:1-SNAPSHOT

[INFO] gdata-docs:1-SNAPSHOT

[INFO] samples-1-SNAPSHOT

[INFO] sample-util-1-SNAPSHOT

[INFO] sample-appsforyourdomain-client-1-SNAPSHOT

[INFO] sample-authsub-webapp-1-SNAPSHOT

[INFO] sample-blogger-client-1-SNAPSHOT

[INFO] sample-calendar-client-1-SNAPSHOT

[INFO] sample-codesearch-client-1-SNAPSHOT

[INFO] sample-photos-client-1-SNAPSHOT

[INFO] sample-tester-client-1-SNAPSHOT

 

[INFO] ————————————————————————

[INFO] Reactor Summary:

[INFO] ————————————————————————

[INFO] google-gdata-mavenized-1-SNAPSHOT ………………… SUCCESS [11.711s]

[INFO] google-gdata-apis:1-SNAPSHOT …………………….. SUCCESS [0.152s]

[INFO] gdata-client:1-SNAPSHOT …………………………. SUCCESS [1:02.902s]

[INFO] gdata-appsforyourdomain:1-SNAPSHOT ……………….. SUCCESS [2.881s]

[INFO] gdata-base:1-SNAPSHOT …………………………… SUCCESS [2.021s]

[INFO] gdata-calendar:1-SNAPSHOT ……………………….. SUCCESS [2.365s]

[INFO] gdata-codesearch-1-SNAPSHOT ……………………… SUCCESS [1.397s]

[INFO] gdata-photos:1-SNAPSHOT …………………………. SUCCESS [3.297s]

[INFO] gdata-spreadsheet:1-SNAPSHOT …………………….. SUCCESS [1.912s]

[INFO] gdata-docs:1-SNAPSHOT …………………………… SUCCESS [1.773s]

[INFO] samples-1-SNAPSHOT ……………………………… SUCCESS [0.173s]

[INFO] sample-util-1-SNAPSHOT ………………………….. SUCCESS [1.304s]

[INFO] sample-appsforyourdomain-client-1-SNAPSHOT ………… SUCCESS [1.651s]

[INFO] sample-authsub-webapp-1-SNAPSHOT …………………. SUCCESS [24.340s]

[INFO] sample-blogger-client-1-SNAPSHOT …………………. SUCCESS [1.466s]

[INFO] sample-calendar-client-1-SNAPSHOT ………………… SUCCESS [0.355s]

[INFO] sample-codesearch-client-1-SNAPSHOT ………………. SUCCESS [1.396s]

[INFO] sample-photos-client-1-SNAPSHOT ………………….. SUCCESS [1.192s]

[INFO] sample-tester-client-1-SNAPSHOT ………………….. SUCCESS [1.656s]

[INFO] ————————————————————————

[INFO] ————————————————————————

[INFO] BUILD SUCCESSFUL

[INFO] ————————————————————————

[INFO] Total time: 2 minutes 6 seconds

[INFO] Finished at: Wed Apr 15 12:40:25 BST 2009

[INFO] Final Memory: 19M/45M

[INFO] ————————————————————————

rus@builder:/proxy/rus/gdata-mavenized$ find . -name *.jar

./gdata-samples/codesearch/target/sample-codesearch-client-1-SNAPSHOT.jar

./gdata-samples/authsub/target/authsub_sample/WEB-INF/lib/gdata-calendar-1-SNAPSHOT.jar

./gdata-samples/authsub/target/authsub_sample/WEB-INF/lib/gdata-client-1-SNAPSHOT.jar

./gdata-samples/appsforyourdomain/target/sample-appsforyourdomain-client-1-SNAPSHOT.jar

./gdata-samples/util/target/sample-util-1-SNAPSHOT.jar

./gdata-samples/tester/target/sample-tester-client-1-SNAPSHOT.jar

./gdata-samples/calendar/target/sample-calendar-client-1-SNAPSHOT.jar

./gdata-samples/photos/target/sample-photos-client-1-SNAPSHOT.jar

./gdata-samples/blogger/target/sample-blogger-client-1-SNAPSHOT.jar

./gdata-apis/codesearch/target/gdata-codesearch-1.0.jar

./gdata-apis/appsforyourdomain/target/gdata-appsforyourdomain-1.0.jar

./gdata-apis/base/target/gdata-base-1.0.jar

./gdata-apis/spreadsheet/target/gdata-spreadsheet-1.0.jar

./gdata-apis/docs/target/gdata-docs-1.0.jar

./gdata-apis/calendar/target/gdata-calendar-1.0.jar

./gdata-apis/client/target/gdata-client-1.0.jar

./gdata-apis/photos/target/gdata-photos-1.0.jar

./gdata-external/java/sample/codesearch/lib/CodeSearchClient.jar

./gdata-external/java/sample/appsforyourdomain/lib/AppsForYourDomainClient.jar

./gdata-external/java/sample/util/lib/sample-util.jar

./gdata-external/java/sample/spreadsheet/lib/GUIDemo.jar

./gdata-external/java/sample/spreadsheet/lib/IndexClient.jar

./gdata-external/java/sample/spreadsheet/lib/ImportClient.jar

./gdata-external/java/sample/spreadsheet/lib/ListDemo.jar

./gdata-external/java/sample/spreadsheet/lib/CellDemo.jar

./gdata-external/java/sample/tester/lib/TesterClient.jar

./gdata-external/java/sample/gbase/lib/gdata-base-cmdline.jar

./gdata-external/java/sample/calendar/lib/CalendarClient.jar

./gdata-external/java/sample/photos/lib/PhotosClient.jar

./gdata-external/java/lib/gdata-client-1.0.jar

./gdata-external/java/lib/gdata-appsforyourdomain-1.0.jar

./gdata-external/java/lib/gdata-spreadsheet-1.0.jar

./gdata-external/java/lib/gdata-base-1.0.jar

./gdata-external/java/lib/gdata-calendar-1.0.jar

./gdata-external/java/lib/gdata-codesearch-1.0.jar

./gdata-external/java/lib/gdata-photos-1.0.jar

Woohoo!

New Ubuntu Jaunty Screen

by rus

At the moment I’m trying out the beta for Ubuntu Jaunty and one the first thing I noticed was the new version of screen available!

Initially you are provided with a menu to choose a theme!

I went with option 3, Ubuntu Dark! You’re then presented with an interesting new screen display, with 2 rows of status at the bottom, in lots of colours, displaying information about the CPU and RAM etc!

There’s a prompt to press F9 for options which presents a curses based interface to change settings and the information on the bottom status bar.

After smiling a little, I decided my best course of action was to delete the screen settings


rus@boosh:~$ rm -rf .screen*

and restart screen using option 1, plain! It’s not that I fear change, but I don’t need to know any of that information! It doesn’t add value to my terminal experience, where as the extra 2 lines of space will!

Generating Secure Passwords With pwgen

by rus

At my current place of employment we have a fairly strict password policy for everything, all passwords must be unique (so no single password used everywhere) and we use KeePass and KeePassX to manage them, so we can have fairly long passwords that aren’t immediately rememberable full of fun characters.

Generating the passwords is a dull process so we take the sting out of it using the pwgen utility!

rus@boosh:~$ pwgen -y 12

Changing variables in a BASH while loop

by rus

There is an error that I often run in to when working with files and while loops in BASH. Often I have scripts similar to the following, where I cat a file and read it in using a while loop to process variables in the file:

#!/bin/bash
count=0;
cat testfile | while read line
do
    count=$(($count+1));
    echo $count;
done
echo "Total $count";

and using the following test data in testfile as:

cake
pie
thongs

I would expect an output like:

1
2
3
Total 3

But instead I get:

idimmu@boosh:~$ ./t.sh
1
2
3
Total 0

What is happening is due to the | (pipe) bash is forking and creating a new process so any variables we are altering and changing ($count) are being manipulated in the child process, and then lost when the subprocess finishes!

A lot of people I’ve spoken to who have seen this either completely change their code structure to accommodate, or worse, change shell completely to something like KSH!

If we tweak our script ever so slightly, and read the file in at the end of the BASH while loop…

#!/bin/bash
count=0;
while read line
do
    count=$(($count+1));
    echo $count;
done < testfile
echo "Total $count";

everything changes 🙂

idimmu@boosh:~$ ./t.sh
1
2
3
Total 3

and we get the output we expect!

bash Cookbook O'Reilly's Bash Cookbook covers everything you'd ever need to know about Bash, including all the nuances of while loops and powerful scripting techniques. Thoroughly recommended.

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

by rus

I’ve been doing a lot of Nagios deployments recently, and this error always bites me, on all Ubuntu versions, including Hardy and Intrepid (haven’t quite bit the bullet to try the Jaunty beta yet 🙂 )


Error: Could not stat() command file '/var/lib/nagios3/rw/nagios.cmd'!

The external command file may be missing, Nagios may not be running, and/or Nagios may not be checking external commands.

An error occurred while attempting to commit your command for processing.

This can be quite easily fixed with the following command line fu:


sudo /etc/init.d/nagios3 stop
sudo dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
sudo dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
sudo /etc/init.d/nagios3 start

Now you should be able to send Nagios remote commands and commands via the web interface to your heart’s content!

Nagios

For more Nagios advice, I recommend Nagios by O’Reilly. It’s full of best practice advice and covers solving more ‘gotchas’ that you might encounter whilst using it!

Burger At The Mash Tun

by rus

For lunch today we all decided to go to the Mash Tun pub. I argued blind that they didn’t do any food, but it turns out, fortunately, that they did!

Burger At The Mash Tun

I had a most fantastic burger, which was almost the size of my pint glass. The patty was really nicely flavoured, with onions and green bits and all sorts, and not over blown with pepper, which seems to be the norm nowadays. The chips were also really good, a lot of places seem to be mimicking the style of Heston’s Thrice Cooked Chips!

Sausage And Mash At The Mash Tun

My friends opted for the Wild Bore Sausage and Mash, which looked and smelled phenomenal.

I would definitely go back again, my only criticism was the waiting time, and lack of out door seats, but you can’t win ’em all!!!

Resize LVM ReiserFS Partition

by rus

Mucking about with LVM and partitions isn’t really taxing, it’s all well documented. Trusting it however is a different matter. I’ve resized loads of Ext3 LVM partitions in the past, but was asked to resize a ReiserFS one today, which made me ask the question:

“Is there anything writing to it currently, as I will have to unmount it first ..”

Wrong assumption, or so says the guy sat next to me! ReiserFS can be dynamically resized on the fly! Woohoo says I, lets have a look at the resize_reiserfs man page!


DESCRIPTION
The resize_reiserfs tool resizes an unmounted reiserfs file system.

“Are you sure?” I ask .. “Yes”, he promises!


anise:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-lv_slot_bbocp
10G 6.0G 4.1G 60% /opt/sem/slot/bbocp
anise:~# lvdisplay /dev/vg00/lv_slot_bbocp
--- Logical volume ---
LV Name /dev/vg00/lv_slot_bbocp
VG Name vg00
LV UUID P9nb4N-ED0Q-jSWh-xLbs-zP72-5QXL-DbyHiM
LV Write Access read/write
LV Status available
# open 2
LV Size 10.00 GB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors 0
Block device 253:11
anise:~# vgdisplay vg00
--- Volume group ---
VG Name vg00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 35
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 13
Open LV 12
Max PV 0
Cur PV 1
Act PV 1
VG Size 219.83 GB
PE Size 4.00 MB
Total PE 56276
Alloc PE / Size 53762 / 210.01 GB
Free PE / Size 2514 / 9.82 GB
VG UUID HDU1Dc-5i7l-wyGE-cEdM-YpdG-kskb-nw0JWm

There is 9GB of free space in the volume group, and our partition wants to grow by 5GB so thats Ok! Lets do this!


anise:~# lvextend -L+5G /dev/vg00/lv_slot_bbocp
Extending logical volume lv_slot_bbocp to 15.00 GB
Logical volume lv_slot_bbocp successfully resized
anise:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-lv_slot_bbocp
10G 6.0G 4.1G 60% /opt/sem/slot/bbocp

No change yet though! Lets do the resize, with it still mounted!!


anise:~# resize_reiserfs -f /dev/vg00/lv_slot_bbocp
resize_reiserfs 3.6.19 (2003 www.namesys.com)
resize_reiserfs: On-line resizing finished successfully.
anise:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg00-lv_slot_bbocp
15G 6.0G 9.1G 40% /opt/sem/slot/bbocp

Woo hoo! We did it, well it’s 15GB in size, and apparently all the data is still there and working and not corrupt. Rejoice. Someone really needs to update that man page!!!

Tomcat HelloWorld Servlet with Eclipse

by rus

I’m really trying to get in to this whole Java web development frame of mind, as it’s a bit of fun, a bit of a giggle, and it’s massive in this area of the world! So obviously my first port of call was dusting off Eclipse and kicking out a HelloWorld style Java servlet!

I grabbed a copy of O’Rielly’s Java Servlet Programming to get started and found it really invaluable, and definitely recommend this book to anyone starting out in Java servlet programming.

Eclipse on Ubuntu, even Intrepid is well old, so rather than work with the out of date supplied package, I found it best to download the latest version direct from the site. Also the easiest way to hook Tomcat in to Eclipse is also to download that from the site.

Get the latest Eclipse IDE for Java EE Developers and Tomcat 6

Extract them both to somewhere reasonable, I like ~/local :


idimmu@boosh:~/local$ ls -al
total 173212
drwxr-xr-x 4 idimmu idimmu 4096 2009-03-11 15:01 .
drwxr-xr-x 63 idimmu idimmu 4096 2009-03-11 15:01 ..
drwxr-xr-x 9 idimmu idimmu 4096 2009-03-11 15:01 apache-tomcat-6.0.18
-rw-r--r-- 1 idimmu idimmu 6142197 2009-03-11 11:27 apache-tomcat-6.0.18.tar.gz
drwxr-sr-x 9 idimmu idimmu 4096 2009-02-23 19:36 eclipse
-rw-r--r-- 1 idimmu idimmu 171022452 2009-03-11 10:57 eclipse-jee-ganymede-SR2-linux-gtk.tar.gz

Start Eclipse! The first thing that Eclipse will do is ask you to create a new Workspace. Your home directory isn’t a bad choice to put this!

Tomcat HelloWorld Servlet with Eclipse

Go to New->Project

Tomcat HelloWorld Servlet with Eclipse

Select ‘Dynamic Web Project’

Set ‘Project Name’ to ‘helloworld’

Create a ‘New’ ‘Target Runtime’

Select ‘Apache Tomcat v6.0’

Select the Tomcat Installation Directory you extracted Tomcat to earlier.

Then click ‘Finish’ to create the project.
Go to File->New->Servlet

Enter ‘HelloWorld’ as the ‘Class name’
Click ‘Finish’

A new ‘HelloWorld.java’ file will be created with most of the work done for you!

Tomcat HelloWorld Servlet with Eclipse

Look at all the shiny code the IDE has already written for you!

import java.io.IOException;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
* Servlet implementation class HelloWorld
*/
public class HelloWorld extends HttpServlet {
private static final long serialVersionUID = 1L;

/**
* Default constructor.
*/
public HelloWorld() {
// TODO Auto-generated constructor stub
}

/**
* @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
}

/**
* @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
*/
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
}

}

This code does nothing on it’s own, well, it will generate a completely empty web page if you build it and deploy it to Tomcat, not very exciting .. so ..

Import the following new class:


import java.io.PrintWriter;

Insert the following code in to the doGet function stub:


response.setContentType("text/html");
PrintWriter pw = response.getWriter();
pw.println("");
pw.println("");
pw.println("");
pw.println("

Hello World

");
pw.println("");

The complete source code for the class will now look like this:

import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
* Servlet implementation class HelloWorld
*/
public class HelloWorld extends HttpServlet {
private static final long serialVersionUID = 1L;

/**
* @see HttpServlet#HttpServlet()
*/
public HelloWorld() {
super();
// TODO Auto-generated constructor stub
}

/**
* @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
response.setContentType("text/html");
PrintWriter pw = response.getWriter();
pw.println("");
pw.println("");
pw.println("");
pw.println("

Hello World

");
pw.println("");
}

/**
* @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
*/
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
}

}

‘Save All’ using Shift+Ctrl+S or the File menu.
Go to the ‘Run’ menu and select ‘Run’ or press Ctrl+F11 to build the servlet, deploy it to Tomcat and run it!

Tomcat HelloWorld Servlet with Eclipse

Make sure ‘Tomcat v6.0 Server’ is selected and click ‘Always use this server when running this project’ then click ‘Finish’

Tada, Hello World! You might have to run it a few times to get Tomcat to sort itself out, as it’s a bit wonky, but the very mundane site should now be available on https://localhost:8080/helloworld/HelloWorld!!

Eclipse

If you really want to be a rock and roll Java super star and learn how to use Eclipse and Tomcat properly, I thoroughly suggest you get a copy of O’Reilly’s Eclipse book and learn how to use it properly.

Nexus on Tomcat 5.5 on Ubuntu Hardy

by rus

I’m trying out this Continuous Integration fun at the moment.

My end game is to get Hudson, Maven and Nexus working together to continuously build and run unit tests against code, which then gets turned in to Deb packages. A new Xen VM will then be created and configured using Puppet which the new Deb package is then deployed to. Finally Selenium will then be run to automate testing of the deployment.

Thats the plan anyway ..

I’ve been deploying everything on Ubuntu Hardy for the time being, and the latest app I am working on is Nexus. I’m deploying it as a War under Tomcat 5.5 and for a while was just getting the following error:


06-Mar-2009 20:29:08 org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive nexus.war
06-Mar-2009 20:29:12 org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
06-Mar-2009 20:29:12 org.apache.catalina.core.StandardContext start
SEVERE: Context [/nexus] startup failed due to previous errors

I hate that error. It’s rubbish. To make it a little more verbose I edited /var/lib/tomcat5.5/webapps/nexus/WEB-INF/log4j.properties and changed the rootLogger to DEBUG and fixed the appender path to go somewhere sensible (/var/log/tomcat55/nexus.log) and restarted Tomcat! This logged the following interesting error:


2009-03-06 20:42:01.066 ERROR [main:] - org.sonatype.nexus.configuration.application.source.ApplicationConfigurationSource:file:
******************************************************************************
* Could not create configuration file [ /usr/share/tomcat5.5/sonatype-work/nexus/conf/nexus.xml]!!!! *
* Nexus cannot start properly until the process has read+write permissions to this folder *
******************************************************************************
2009-03-06 20:42:01.091 ERROR [main:] - org.sonatype.nexus.Nexus:default: Could not start Nexus, bad IO exception!
java.io.FileNotFoundException: /usr/share/tomcat5.5/sonatype-work/nexus/conf/nexus.xml (No such file or directory)

Not exactly rocket science to see what is going on here!! And really easy to fix!


mkdir /usr/share/tomcat5.5/sonatype-work
chown tomcat55: /usr/share/tomcat5.5/sonatype-work

A quick restart of tomcat, after turning the logging back down to INFO and ta da! A working Nexus repo!

BGP: Building Reliable Networks with the BGP

by rus

I recently had to learn BGP in order to fully understand what I was doing when migrating from our Linux based Zebra routers (crappy and unstable) to our nice shiny new Juniper routers.

We had a couple of books at work, but by far the best was BGP: Building Reliable Networks with the Border Gateway Protocol. It hasn’t changed much since it was released in 2003, but neither has BGP so that really isn’t an issue, and with the recent BGP issues that keep occuring, a good knowledge of it is pretty important to any network admins out there, especially those like me with their own AS numbers!

The book is full of real world examples and was all I needed to use in my migration.

The book is a bit of a bore to read though, it’s not a comic, or a fun adventure book with a twist at the end. It’s a solid information transmission system which will inject in to your brain how the many networks that make up the Internet actually talk to each other and how traffic flows between them, which is exactly what I wanted and why I give it a solid 8/10.

The latest update, now just called BGP, is a must have if you want to start running the border gateway protocol and start broadcasting your own address range. You don’t accidentally want to knock Youtube of the Internet like China did.. do you?