Today we did a hero WOD called the Tumilson. The RX is
8 rounds for time of:
Run 200 meters
11 Dumbbell burpee deadlifts, 60 pound dumbbells
But I scaled it with 25kg dumbells ~55lbs.
collectd illegal attempt to update using time when last update time is (minimum one second step)
Whilst debugging our CollectD and Graphite setup, I found that CollectD was generating hundreds of repeated errors.
Jun 13 17:11:14 hmon collectd[15336]: rrdtool plugin: rrd_update_r (/var/lib/collectd/rrd/hmon/processes/ps_state-running.rrd) failed: /var/lib/collectd/rrd/hmon/processes/ps_state-running.rrd: illegal attempt to update using time 1371136272 when last update time is 1371136272 (minimum one second step)
Jun 13 17:11:14 hmon collectd[15336]: rrdtool plugin: rrd_update_r (/var/lib/collectd/rrd/hmon/processes/ps_state-sleeping.rrd) failed: /var/lib/collectd/rrd/hmon/processes/ps_state-sleeping.rrd: illegal attempt to update using time 1371136272 when last update time is 1371136272 (minimum one second step)
Jun 13 17:11:14 hmon collectd[15336]: rrdtool plugin: rrd_update_r (/var/lib/collectd/rrd/hmon/processes/ps_state-zombies.rrd) failed: /var/lib/collectd/rrd/hmon/processes/ps_state-zombies.rrd: illegal attempt to update using time 1371136272 when last update time is 1371136272 (minimum one second step)
These messages were occurring every minute or so and filling up /var/log/messages.
A quick search of the Internet shows that unfortunately this is a known issue which will be fixed in a further release of the CollectD 5.1 branch.
We were currently running
collectd-5.1.0-1.el6.rft.x86_64
collectd-rrdtool-5.1.0-1.el6.rft.x86_64
An upgrade to 5.3.0 fixed it!
Bing The Most Altruistic Search Engine
I needed to access Bing’s Webmaster Tools to have a look at the configuration for this site. I couldn’t remember the URL off hand so I decided to ‘Google’ the search term using Bing!
That’s very nice of them … I guess part of Bing’s preference engine might rank pages higher depending on where the search words are in the title of your page or something!
Think Carefully About Your Clever Project Names
We’re building a new exciting cluster at work using Linux HA and stuff to make it work magically.
In the olden days of yore RedHat and co were using Pacemaker with the old crm (Cluster Resource Manager / Cluster Relationship Manager, pick one..) tool for cluster management, which was nice. Now it looks like RedHat have removed the crm command from their repositories and have switched to PCS which stands for either Pacemaker/Corosync Configuration System or is the plural of PC (Personal Computers).
This makes searching for anything useful on the Internet quite difficult.
It would have been better to go with a random pronouncable alpha-character string non word, than the stupid PCS acronym, and probably get you some delicious Web 2.0 VC!
Income Report March 2013
Following on from last month’s report March was quite an interesting month. Changing perspective from using this site as a simple memory dump to one that people, other than myself, would find useful lead to some interesting usage patterns, from both myself and my visitors as well as some changes to the blog!
Due to my change of attitude I actually watched my traffic stats, where I noticed some odd things in my logs, I thought I fixed the problem… It turns out having an URL or page with the phrase ‘apache access logs’ in it is a really bad thing as now that page is plagued with bot hits using Russian domains as a referrer, trying to get a back link for their domain in my web stats. It’s receiving about 1500 unique hits a day at the moment which is skewing my stats quite heavily.
As an experiment I increased the size of the Adsense banners to their larger size, which fit in my new theme. I also attempted to monetize some of the more applicable pages with affiliate links back to O’Reilly products. I think I’m going to have to start to use the number of Adsense views as an indicator of traffic, rather than awstats from now on, due to these pesky Russians. I hope they aren’t mafia!
March 2013 Income Report
Adsense Page Views: 5227
Adsense Clicks: 19
Adsense: £7.50
O’Reilly Page Views: 7022
O’Reilly Clicks: 85
O’Reilly: £0.00
So that’s a really interesting increase, up from £0.33 last month, purely by changing the size of the Adsense banner. Awstats is reporting 8361 unique visitors, up from 6332 in February, but unfortunately I can’t trust that metric any more due to the heavy bot activity skewing the stats. Adsense is reporting 3,889 page views for February so it does look like I am actually getting more traffic.
After speaking to the owner of Lampdocs after a recent post they made about trying to monetize a Linux blog, the main conclusion we came to was more traffic and Adsense. Unfortunately my strength comes from my personal systems administration and development skills, I wonder if I can come up with a killer product worth selling ..
Installing Magento
One of my clients wanted an E-Commerce solution for his website and after a little bit of analysis we opted for the community edition of Magento.
We wanted something based on the usual LAMP stack, that was OpenSource so it could be extended, was free as in beer, had great international support, was fully featured and enterprise ready, could pass PCI DSS compliance, allowed reasonable payment gateway options, could scale and was easy to extend as well as backup. Also we wanted complete control of the deployment, rather than integrating with a 3rd party cloud service provider such as Shopify to keep costs down and retain flexibility.
Installing Magento
With out going in to the specifics of configuring an Apache VirtualHost or installing MySQL here’s a rough guide on how to install Magento.
Download Magento from their download page. Always opt for the latest version as it includes important security fixes. You will need to create an account on their site for this.
Create a new MySQL database for the installation and note it’s credentials for later use.
Extract the Magento archive to your document root.
You will need to set write permissions for the web server to write to the following files and directories.
- var
- var/.htaccess
- app/etc
You can either chmod 777 or get a little cleverer about ownership. e.g.
sudo chmod -R 777 var var/.htaccess app/etc
Assuming Apache is correctly configured you can now use your web browser to visit the install directory at the URL you’ve installed Magento at and follow the online install guide. e.g. https://www.shopidimmu.net/, then the wizard will be located here: https://www.shopidimmu.net/install/.
Choosing A Credit Card Processor
After we’d correctly installed and configured Magento, I left it to the web team to get on with populating it with products and get it to look pretty but we still had to find the best credit card processor to accept payments as we didn’t want to use Paypal. We also wanted someone who offered physical solutions to accept card payments with chip and pin.
After checking out some reviews for Merchant Warehouse and Charge.com we decided to opt with Merchant Warehouse as they offer really easy integration with Magento with their MerchantWare plugin.
Installing the Magento Connect MerchantWare plugin was trivial and just required copying and pasting our extension key in to the Magento Connect Manager and clicking install.
monit: error connecting to the monit daemon
We’re rolling out monit on our new platform at the request of a vendor to manage their new service. I’ve always been dead against these kinds of automated failure recovery tools as they often require human intervention after the fact anyway and all the platforms I’ve managed will have failed the server anyway so why not restart the services after the root cause analysis is done? My tune is slowly changing though and I’m coming to appreciate this method of systems recovery a lot more.
Whilst playing with it though I got the following error
root@newshiny:~# monit summary
monit: error connecting to the monit daemon
What what? The daemon’s definitely running, why can’t I pole it’s status?
root@newshiny:~# ps aux | grep monit
root 325293 0.0 0.0 16440 1276 ? Sl Mar27 0:10 /usr/bin/monit
root 496627 0.0 0.0 105348 832 pts/0 S+ 11:43 0:00 grep monit
root@newshiny:~# service monit status
monit (pid 325293) is running...
After reading the documentation this monit: error connecting to the monit daemon seemed to be an epic case of rushing in to things, skimming the documentation and PEBCAK!
Solving monit: error connecting to the monit daemon
Monit can present an HTTP interface which I didn’t enable as I thought it was just for me, it turns out it’s also for the command line tools!
It’s really easy to enable, in /etc/monit.conf or wherever your conf file is located just add
set httpd port 2812 and
use address localhost
allow localhost
and restart monit with
service monit restart
and Bob’s your mother’s brother.
root@newshiny:~# netstat -lpn | grep 2812
tcp 0 0 127.0.0.1:2812 0.0.0.0:* LISTEN 325293/monit
root@newshiny:~# monit summary
The Monit daemon 5.2.5 uptime: 19h 18m
Process 'shiny_manager' running
Process 'shiny_proxy' running
Process 'shiny_server' running
System 'system_newshiny' running
Problem solved!
Odd Occurrences In Apache Access Logs
I’ve been watching my Awstats installation this month as idimmu.net is about to peak with over 7000 unique visitors in a month for the first time in it’s history, which is pretty awesome. But there’s been something really weird going on in the results ..
This is kind of ironic as in a recent job interview I was asked
What would you look for to ascertain suspicious activity on an instance of Apache serving static image assets?
Obviously I aced the question, and whilst my server isn’t limited to static assets, it does have the GNU tool chain installed 😉
6487 views for my Elgg CSS Fix page this month. For the record at the time of writing the site has had 6908 unique views and the next highest viewed page is only at 2446 views. Something is up! I wonder what ..
root@holly /var/log/apache2 # grep elgg idimmu.net.access.log | head
91.198.94.225 - - [24/Mar/2013:06:44:52 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "https://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:44:52 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "https://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:44:53 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "https://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:44:53 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "https://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:54 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "https://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:54 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37928 "https://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:54 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "https://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
91.198.94.225 - - [24/Mar/2013:06:47:55 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "https://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
216.152.249.243 - - [24/Mar/2013:06:49:34 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.1" 200 9931 "https://www.idimmu.net/" "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.0) Opera 7.02 Bork-edition [en]"
91.198.94.225 - - [24/Mar/2013:06:50:55 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/ HTTP/1.0" 200 37927 "https://www.idimmu.net/2011/11/21/elgg-1-8-tidypics-group-fix/" "Opera/9.80 (Windows NT 6.2; WOW64) Presto/2.12.388 Version/12.11"
Ok, so my access log rotated out on the 24th March but the data is still meaningful with out having to put together exact logs for the entire month! One IP 91.198.94.225 seems to be retrieving the Elgg page over and over again, several times a minute?!
root@holly /var/log/apache2 # grep 91.198.94.225 idimmu.net.access.log | wc -l
5876
root@holly /var/log/apache2 # grep elgg idimmu.net.access.log | grep 91.198.94.225 | wc -l
5876
And seemingly that same IP address is ONLY requesting my Elgg page, no other pages!
root@holly /var/log/apache2 # grep index.php idimmu.net.access.log | grep 91.198.94.225 | head -n 1
91.198.94.225 - - [24/Mar/2013:06:44:53 +0000] "GET /2011/11/21/elgg-1-8-tidypics-group-fix/index.php HTTP/1.0" 301 471 "https://www.idimmu.net/index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11"
root@holly /var/log/apache2 # grep index.php idimmu.net.access.log | grep 91.198.94.225 | wc -l
2938
root@holly /var/log/apache2 # grep -v index.php idimmu.net.access.log | grep 91.198.94.225 | wc -l
2938
Also half of it’s requests are for _/2011/11/21/elgg-1-8-tidypics-group-fix/index.php_ and the other half are for _/2011/11/21/elgg-1-8-tidypics-group-fix_ which is double weird. The index.php page extension just redirects to the extension-less URL due to the incredible intellectual artificial intelligence running the CMS platform I use!
The Elgg page has maybe 1 or 2 back links out there, none are to the index.php extended URL and looking at the Apache logs the referrer is https://www.idimmu.net/index.php which isn’t a valid URL anyway, which suggests whatever is doing this is spoofing the referrer.
Also also, what kind of browser definition is
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1.0 Safari/537.11
Some random infosec page I’ve never heard of suggests it’s a valid Chrome user agent running on Windows. You can write lots of interesting plugins for Chrome, including scrapers and bots so this is possibly a valid option. Why it also mentions Safari I’ve no idea though!
The 91.198.94.225 IP appears in Google several times for spamming so possibly that’s the point of the bot. Judging by the rest of the search results there appears to be a LOT of comment spam to old PHP guestbook implementations that look to have seemingly trivial captures, so maybe the bot thinks my site is running one of these? It does appear to be stuck in some kind of infinite loop on one page however 🙁
Whois says it’s an IP located in Poland! Has anyone else seen any ‘interesting’ impressions from this IP address, or anything similar? Fortunately I’m pretty good with computers.
iptables -A INPUT -s 91.198.94.225 -j DROP
In your face Polish!
For more iptables related nonsense I suggest you try O’Reilly’s Linux iptables Pocket Reference or man iptables if you don’t want to buy anything 😉 Or you know, just keep reading my stuff!
Experiments In Bitcoin
Earlier today I got in to discussing Bitcoin Arbitrage with a mate of mine, with the cunning plan of creating a bot to monitor different exchanges and profit on the difference in trading prices. After an analysis this was deemed a silly idea, or conversely we aren’t good enough, as moving real money or bitcoins around the exchanges seems to be an extremely painful process. The first step though was creating a wallet!
Bitcoind
Multibit is a fantastic cross platform GUI Bitcoin wallet which is super easy to use but limits you to only using the workstation you’ve installed your wallet on, unless you get all remote desktopy. There are also numerous places that let you host your own wallets and most exchanges also feature a wallet too, but I’m very DIY! BitcoinD however is a little cross platform daemon that just sits their ticking away, managing your wallet, whislt responding to JSON-RPC over HTTP. Perfect for me!
‘Installation’ on Linux was easy, just download the tar.gz, extract, then run the bitcoind binary of choice! I run mine in a screen but there are init scripts if that’s your thing. It creates a wallet file in ~/.bitcoin. When it starts it suggests you configure a username and password for the RPC interface. You should do this. My ~/.bitcoin/bitcoin.conf looks like this:
rpcuser=rusisawesome
rpcpassword=thisismysecretpassword
GUI
Ok, so you’ve a deamon running, now what. Well it logs to ~/.bitcoin/debug.log which is some indicator that it’s working, but a pretty GUI would be nice. I opted for bitcoin-js-remote, a nice javascript web app app, that comes with it’s own daemon or can be served over Apache. I’ve got Apache installed at home so I opted for that.
Extract bitcoin-js-remote somewhere nice, e.g. /var/www/bjr and configure Apache. You’ll want to use HTTP Basic Auth to protect the directory, which we’ll go in to, and ideally use SSL (best practice ftw) which we won’t go in to!
You’ll need to create, or add to, an htpasswd file with the RPC credentials you’ve configured bitcoind to use earlier.
htpasswd -b -c /etc/apache2/htpasswd.pass rusisawesome thisismysecretpassword
BitcoinD by default will listen on port 8332 on localhost, and the bitcoin-js-remote will call back via Apache to talk to it so we need to configure Apache to do a bit of proxying as discussed here.
AuthType Basic
AuthName Restricted
AuthUserFile /etc/apache2/htpasswd.pass
Require valid-user
ProxyPass /lbc/ https://127.0.0.1:8332/
AuthType Basic
AuthName “Bitcoin Access”
AuthUserFile /etc/apache2/htpasswd.pass
Require user rusisawesome
Order deny,allow
Allow from all
Restart Apache and jobs a good’n.
Bitcoin-js-remote also has a settings.json file. I’m not sure if it’s necessary but I added my credentials to that:
{
"RPC": {
"url": "/lbc/",
"user": "rusisawesome",
"password": "thisismysecretpassword",
"account": ""
},
"labelsmode": 1
}
I don’t think this is necessary to be honest. You can now connect to your webserver, present the basic auth credentials and it should log you straight in to your new shiny web based client!
If you are hosting your own wallet, it’s advisable to back it up, as if you lose it, you lose all the money that’s inside it!
There are some free sources on the Internet that you can use to get some meagre BTC to play around with, I think I’ve got around $0.15 worth, though as you can see I’m still waiting for transactions to come through, so far the fastest has taken 3 hours and the slowest over 12 hours. Moving BTC around different exchanges for arbitrage purposes is kind of hopeless …
My Bitcoin address is 13gcobY9ofgnokCsU436yNVgCqGB3PVW6z, there are even nice bitcoin URL shortners so I don’t have to remember that monstrosity!
The Blockchain site, as well as offering wallet hosting, lets you observe and track all the transactions going on in the Bitcoin network as well as see the state of all the wallets on the network (if you know the address). The details of my wallet are here and the first transaction made to it earlier today is here. I have to say when I saw that transaction I got all excited by the prospect of almost 5 BTC, but it appears that multiple transactions can be done in the same block, mine was the smaller one at 0.002 BTC 😉
I ran a Bitcoin miner on my MacBook at work earlier and it reckoned it would take 10,000 days to generate 1BTC. I think I’ve missed the bubble there 😉
At the moment Bitcoin feels a little like the Esperanto of the financial world, but you can already by drugs and pay for strippers and pizza with it, which is probably more than Esperanto will let you do!
What’s really awesome about bitcoin is the bitcoin community itself. Reading about it feels like there’s a lot of entrepreneurial adventure going on whilst everyone is making land grabs to provide previously taken for granted services in new and exciting ways due to the financial flexibility that bitcoin provides. I think it’s exciting.
Android Create AVD
The second part of this series covers creating our first Android Virtual Device or AVD. The Android Virtual Device (AVD) is basically a configuration for the Android SDK’s emulator that lets you define the hardware and software characteristics of an actual Android advice that you can then test your code on. Whilst not being as good as running your app on real hardware, testing it on an AVD is the next best thing!
You can either watch the Android Create AVD video above or follow the steps below, it’s dead easy!
First we need to start the AVD Manager by going to Window -> AVD Manager
Then click on New to bring up the AVD Configuration Dialog.
Enter an appropriate name, select your API target, the size of the SD card and skin/resolution you want to use then click Create AVD.
The AVD creation will take about 1 minute to run, but then the new AVD will appear in your list of available AVDs. It’s possible to make many different AVDs to test your application on many different screen sizes and hardware profiles!
If you’re new to the Android mobile operating system, Learning Android is the perfect way to master the fundamentals. This gentle introduction shows you how to use Android’s basic building blocks to develop user interfaces, store data, and more. You’ll build an example application throughout the course of book, adding new features with each chapter. You’ll also build your own toolbox of code patterns that will help you program any type of Android application with ease.