Dev Thoughts

For the past few months I’ve been working on a startup, something very useful that fills in a need that businesses have.

Right now I’m deep into adding the finishing touches on it so that it can be released for consumption. This does not mean that it’s finished by any stretch of the imagination – I have a todo list that’s as long as a piece of string – but it does mean that we’ve crossed enough t’s and dotted a sufficient number of i’s so that other people can use the website.

If all goes well the site proper should be up and live in a few days at the start of February, and I’ll do another post then.

Stay tuned!

I have been using the Google search engine for many years now, and have found it to produce good search results, that is until recently. What’s been annoying me now is that I do frequent searches to find information related to fields that change frequently, such as Ruby and Rails, and the results that I get are the oldest ones which are no longer valid or are innacurate.

Generally when I search for a particular problem I am having, the first results that come up have been from a blog that is 2-3 years old, and about a dozen versions out of date. Though it’s not only Google that has this problem, but a lot of the other major search engines. It’s that they give more precedence to how old a particular site is rather than if it’s relevant anymore. This also disproves the adage that there’s no more innovation to be had in search.

For now though what would be really useful would be the ability to sort the search results by date. From newest to oldest, oldest to newest, so that the information for the newest versions would come out on top rather than be buried in the search results. I’d love to see this as an advanced search option, or even one of those Google search box commands.

Recently I’ve developed a site using the simple StaticMatic website generator. StaticMatic is a nice and simple static website generator that uses HAML and SASS to describe the site content while also providing an easy way of using templates much like in Rails. However one thing that it does not do is provide an easy way of doing deployment to a server.

Looking over the deployment options I decided to use Capistrano, mainly as a learning experience before trying to use it to deploy proper Ruby on Rails applications. Though I found out that it’s not that difficult to set up. So here goes my story:

Installation

To deploy StaticMatic via Capistrano you need to use the railsless-deploy package which cuts out the Rails specific cruft from Capistrano.

You might need to add http://gems.github.com (I know they don’t do Ruby gems anymore, but it still works for me) and http://gemcutter.org to your gem sources list. To do this use:

gem sources -a <SOURCE>

Then installing the required packages is simple:

gem install capistrano
gem install railsless-deploy

Setup

First you’ll need to Capify the site, so in the StaticMatic site root directory execute:

capify .

Which will create the Capfile and config/deploy.rb files. These now need to be edited to work with railsless-deploy and StaticMatic.

So the Capfile with the appropriate modifications should look something like:

load 'deploy' if respond_to?(:namespace) # cap2 differentiator

require 'rubygems'
require 'railsless-deploy' # Removes railsisms from Capistrano
load 'config/deploy'

Then the config/deploy.rb needs to be worked on. This is where you set up the configuration for how to deploy the site. There are many settings that need to be set, such as the application, the repository, and the roles available (I just set the roles to the same thing). In addition I use git for source control of this site, so some of my settings are:

set :deploy_to, "/var/www/#{application}"
set :deploy_via, :export
set :scm, :git
set :shared_children, []

Using the export deployment method means that there is no .git directory hanging around on the server for people to muck around with. Setting the shared_children variable to an empty array means that no shared directories are present on the site. While this isn’t essential, I use it so that no superfluous directories are created.

In addition to these settings, a couple of small task definitions need to be included in the config/deploy.rb file, namely:

namespace :deploy do
  task :update do
    transaction do
      update_code
      build_code
      symlink
    end
  end

  task :build_code, :except => { :no_release => true } do
    run "staticmatic build #{latest_release}"
  end
end

These tasks are necessary to build the site on the server, with the first task redefining the deployment task so that the code is built at the appropriate stage, and the second task doing the actual building of the site.

Deploying

Once the site has been configured appropriately a few Capistrano commands need to be executed to set up the environment on the server. Firstly the directory structure needs to be set up, so run:

cap deploy:setup

Then we check that everything is configured appropriately by running:

cap deploy:check

And finally we can deploy the first version of the site using:

cap deploy

Assumptions

Now of course this post isn’t a complete how-to of deploying a StaticMatic site onto a server. Things that I have omitted are setting up a deployment user on the remote site, generating a SSH key, configuring Capistrano with that user information, etc. However there are various other guides out there with this information so it shouldn’t be too hard to find.

I’ve added a page that contains a list of books that I’ve found interesting from both a personal and professional standpoint. More importantly there are links to both Amazon.com (USA) and Amazon.co.uk (UK) in case anyone wants to buy them, of course I get a tiny cut if people do this.

I’ve been doing some Ruby on Rails development recently and one thing that’s bugged me a bit is how to deal with test data while you’re developing the application. Not data that is used just for testing, but data that lives in the development database that you use and fiddle around with while developing the site itself. I’ve found that while developing features of the site I need a fairly large and varied set of data to use, and this data takes a while to enter into the site each time I modify the database.

So instead of repeatedly entering this data I wrote some simple Rake tasks that can automate exporting and importing the data from the database. I can’t take all the credit as the idea was inspired/based on/stolen from the Advanced Rails Recipes book.

Exporting works by writing each table to its own CSV file in the db/dataset directory (or subdirectory) in the rails project. The first line of the file is the name of the columns in the database.

  desc "Exports the database contents into CSV files."
  task :export => :environment do
    require "FasterCSV"
    require "FileUtils"

    path = File.join(DATASET_PATH, (ENV["DATASET"] || "."))

    FileUtils.mkdir_p(path)

    ActiveRecord::Base.establish_connection
    database = ActiveRecord::Base.connection

    (database.tables - SKIP_TABLES).each do |table_name|
      FasterCSV.open(File.join(path, "#{table_name}.csv"), "w") do |csv|
        # Write column names
        csv << database.columns(table_name).map(&:name)
        # Write rows
        database.select_rows("SELECT * FROM %s" % table_name).each { |row| csv << row }
      end
    end
  end

Importing works much the same way by reading all of the CSV files in the db/dataset directory (or subdirectory) and attempting to load the data into the database. However it does check that the table of that name exists first and skips it otherwise.

  desc "Imports data from CSV files into the database."
  task :import => :environment do
    require "FasterCSV"

    path = File.join(DATASET_PATH, (ENV["DATASET"] || "."))

    ActiveRecord::Base.establish_connection
    database = ActiveRecord::Base.connection
    tables = (database.tables - SKIP_TABLES)

    database.disable_referential_integrity do
      Dir.chdir(path) do
        Dir.glob("*.csv") do |file_name|
          table_name = file_name[/(.+)\.csv/, 1]
          next unless tables.include?(table_name)

          rows = FasterCSV.read(file_name)

          # Construct a list of column headings from the first row
          heading_row = rows.shift.map { |heading| database.quote_column_name(heading) }.join(",")

          rows.each do |row|
            # Need to escape the data correctly so that inserting is successful
            row_data = row.map { |item| database.quote(item) }.join(",")
            database.execute("INSERT INTO %s(%s) VALUES (%s);" % [table_name, heading_row, row_data])
          end
        end
      end
    end
  end

These are faily simple tasks and therefore I’m releasing the code to the public domain, grab the file and use it freely.

To make it work just add it to your lib/tasks directory in your Rails project. The tasks can then be invoked with:

rake dataset:export
rake dataset:import

You can also specify a specific subdirectory by passing the DATASET option to rake, like:

rake dataset:export DATASET="dev"

Of course I know there are many things missing from this script, such as being able to specify the rails environment to use, but I’ve kept it simple so adding these features is left as an exercise to the reader.

I’ve been programming in C++/CLI at work recently, and I thought I’d share a snippet/trick or two that I’ve learnt.

When working with the .Net framework in C++ you can come across places where it’s easier to use the Win32 functions to do a particular task rather than the .Net ones (or there’s no direct .Net version of it available). In those cases you may need to get Win32 handles from some .Net classes, or extract primitive types from a handle to an Object instance.

To get the window handle (HWND) of a .Net Form:

HWND form_handle = reinterpret_cast<HWND>(form.Handle.ToPointer());

This also works for other handle types used in Windows, such as HDC.

Effectively this is just unboxing the variable from the .Net type. However the Handle property returns a type of IntPtr and therefore it needs the ToPointer() call to work properly.

Another trick to proper unboxing is when trying to convert from an Object^ to a primitive type, say UINT, or LONG. I discovered that using a safe_cast directly to the primitive type did not work, instead it just raised an exception. So to correctly unbox primitive variables you need to unbox them to a primitive, non-typedef’d type, something like:

int a = 10;
Object^ a_obj = gcnew Object(a); // Automatic boxing
UINT new_a = safe_cast<int>(a_obj); // Unboxes

Also don’t forget to wrap this unboxing code in a try – catch block as it might still throw exceptions.

It’s funny how things have come full circle in the Windows game development world – when it comes to processing player input of course. It used to be back in the day (I think around 2000 or so) that a game engine would use DirectInput to manage user input, but that has now been deprecated in favour of using plain old native Win32 messages.

(Though you still need to use DirectInput for Joysticks that require it, and XInput for newer devices – such as XBox360 gamepads. There’s an article on how to use both side by side.)

I came to this realisation when working on my own engine recently, as I was planning on coding up a nice primarily DirectInput based system, and only using the native windows functions to handle the mouse input when windowed. However after reading an article or two and looking at some of my old projects I decided against it and just used simple windows message handling to get the job done.

Some of the reasons to no longer used DirectInput are:

• That it isn’t really all that direct at all, it just spawns another thread that listens for the input related windows messages and processes them for you.
• It also doesn’t play nice with the user’s input related settings in the control panel, such as the key repeat rate, and mouse speed and acceleration settings.
• And it doesn’t handle international keyboards as well as the alternative.

Additionally it turns out that not only is using the native windows messages easier to code up, but the code is also shorter and it makes the other architecture cleaner and more elegant to boot.

Technical Details

To use the windows messages as input all you need to do is to process all of the relevant messages:

• WM_KEYDOWN, WM_KEYUP – For the regular keystrokes
• WM_SYSKEYDOWN, WM_SYSKEYUP – For the ALT key, can be processed by the same functions as the regular keystrokes.
• WM_MOUSEMOVE – For the mouse movement
• WM_MOUSEWHEEL – For the mouse wheel (Only on Windows XP and above)
• WM_*BUTTONDOWN, WM_*BUTTONUP – (Where * is L, R, M, X) For mouse buttons
• WM_CAPTURECHANGED – Needed to properly handle some aspects of the mouse in windowed mode.

(There is also another way of getting far more precise mouse values using the more generic input messages, but I did not need this for the project so I did not use it. However developers of first person shooters or other twitch games should investigate this method.)

Then pack the data into your own custom structure so that you don’t have to have the windows message handling values all through your code, and then dispatch the data structure to your game state controller for handling.

Implementation Notes

Now there are a few things to watch out for when creating this system, mainly dealing with the processing of mouse based input, and especially when operating in windowed mode.

Mouse Position

You should cache the mouse position as it is received from the WM_MOUSEMOVE message and only dispatch the new mouse position once per frame in an update function or similar. The reason for this is that the mouse update rate is far higher than the frame rate and movements in the mouse can generate many such messages which flood the system. This is especially bad when expensive computations are performed on the movement of the mouse (such as camera orientation/position) as they will suck up all of the CPU time doing the processing.

When switching the mouse from reporting absolute to relative coordinates there are several things that need to happen. The mouse needs to be captured, the cursor hidden, the current coordinates stored for later, and then set to the centre coordinates of the window. I found that sometimes the mouse cursor show count is greater than one, so I repeatedly call the ShowCursor(FALSE) function to make sure it is hidden. Otherwise the user can still see the mouse as it is wildly moving about the centre of the window. Then at the end when switching back into absolute coordinates don’t forget to restore the previous position of the mouse.

Mouse Wheel

For every mouse that I have encountered the value reported is a multiple of 120 units, with 120 (or WHEEL_DELTA in the code) units representing a single “click” or movement of the mouse wheel. As the Microsoft documentation explains this is so that in future we might have mice that scroll at a finer granularity. Therefore I made sure in the processing that I first accumulated the incoming value and then only reported the movement if it moved more than 120 units.

Additionally I discovered that (at least) in debug mode when I scrolled the mouse wheel I would sometimes get multiples of 120 showing up as the incoming delta. Where as I had not seen this behaviour before at work or otherwise. Therefore it is possible to return more than one “click” or significant movement of the scroll wheel in a single message.

Mouse Buttons

When the user presses down on a mouse button within your client area the mouse should be captured by your application. This is so that you will get a mouse button up message when the button is released while the mouse is no longer above your client window. (That’s also why the WM_CAPTURECHANGED message is listed above.)

Keyboard

One tip for dealing with the keyboard is to only report the first instance of a key down message. You can detect this by checking the status flags on the incoming message (Bit 30 of the wparam parameters I believe, but check the documentation). This blocks the key repeat feature of the messages from going to the rest of your application so you’ll only get one key down event. Hence enable it for the general game but disable it when the user is typing into a text input field, as you’ll want to allow the default system key repeat settings to take effect.

That’s it for now. Hopefully this has helped someone that was in a similar situation to me before.

This is an update on the game development project I am working on as a hobby. Initially the goal was to make an engine and a small game an enter it into the TSumea AI programming challenge. However I wasn’t able to complete the entry in time as I really bit off far more than I could chew.

It came down to the fact that developing an engine, a game, and an interesting AI simulation is far more for one person to do in three weeks of hard coding. There are some questions to consider as to why I didn’t have an entry ready.

• Maybe I could have used an existing open source engine – but none of them gelled with me nicely.
• Maybe I could have started developing the game and engine back in September (when the challenge started) – but there were work deadlines to make.

Regardless, the final result is that I didn’t get the engine or game completed. Not only that, but from all the people that suggested interest in the challenge there was only one final entry. How disappointing.

But still I have developed something that looks like an engine.

In three weeks “working” about 12 to 16 hours a day of programming/research I have managed to create a nice engine, with (what I hope is) a good architecture underneath it. I even spent a weekend working out mathematical and physics formulae (for collision detection and response) on my Tablet PC, and I can honestly say that I haven’t done that much mathematics since University.

(I have to note that it’s quite nice to scribble out mathematics on a tablet. As you can use normal handwriting with the correct mathematical symbols and all on a grid ruled background. Also you can move stuff around and erase with ease if you make a mistake.)

For the time spent on the project I have had a blast. It reminded me of the good old Uni days staying up till the early hours in the morning coding, and yes I did that this time as well. I learnt to use some interesting bits of the Boost library, experimented with better ways of writing game architecture code, and re-learnt a whole bunch of mathematics and phyics.

However the engine and game is not complete, and since I stopped working on it on the 15th I haven’t done much to it since. (I’ve been playing games instead!) But I’ll start working on it again now and see if I can create a decent game out of it. Also I have a couple of articles that are in various stages or preparation and will post them when they are done.

Have a happy new year too!

Now that I’ve actually been using Visual Studio for a lot of development work at home I’ve encountered some annoyances. One of these has been that sometimes when switching between windows using the Ctrl+Tab/Ctrl+Shift+Tab shortcuts it gets stuck and refuses to take any more keyboard input until I click on it with the mouse.

A quick glance at the problem reveals that it’s a focus issue with the window that pops up listing the open documents, where if you switch quickly enough it doesn’t get time to properly return focus to the main window.

Luckily there’s a handy solution out there that fixes the problem, you simply swap what the Ctrl+Tab and Ctrl+F6 keyboard shortcuts are bound to (but don’t forget the Shift variants).

I get really annoyed by obtrusive software update mechanisms. They steal focus, popup dialogs, show message boxes, add random icons to your system try, and then popup balloon items. All in a vain attempt to let you know that some crappy little app that you rarely use has an update available and that you should stop and drop everything that you are doing to install the update right now because it really can’t wait one second longer! Obviously someone involved making the software thought that keeping the little application up to date is a far better use of your time than you doing the things you need/want to do.

The main gripe that I have with software updaters is that they announce the update at completely arbitrary and inopportune times, often covering whatever you were actually trying to do with the program. Or better yet you’re concentrating and getting stuff done and up comes a window asking you to restart the application/computer. And those are just the friendly updaters, some just restart the application right smack bang from under you, destroying whatever you were working on. There are many offenders that I could name, but just off the top of my head I can call out Java, Acrobat Reader, Logitech software, Paint.Net, Apple/iTunes, and Firefox as programs with bad update mechanisms. (Some worse than others of course!)

A particularly special gripe I have for the Windows updating mechanism. The updates do warn you if the computer will need to be restarted after installation, but almost every single update states that it may need to restart my system. So it’s pretty much random as to which updates will force your system to restart and which one’s won’t. The other problem is that the options for how the update system works are all undesirable in my view.

• You can set it to never update, but that just means you need to remember and perform the update process manually. It then either becomes a chore and people get annoyed with it and stop doing it, or they just don’t bother to do it in the first place, and the system is left vulnerable.
• Another option is to get it to notify you when updates are available and you can then elect to install them (the download and notify I class as the same option really), but really it’s annoying as you get both bugged about it, and then have to manually perform the update anyway.
• The last option is to let the system automatically install updated at some point during the day (usually this is at night), however this means that you need to keep your computer on overnight. Whenever I leave my computer on overnight it’s doing something for me (like downloading large files, processing data, etc) so I get annoyed when the system decides to restart itself after an update simply because I wasn’t there to dismiss a dialog in time. (Though there is a fix for this, hurray!)

I know of many instances where people have been either confused or panicked by simple update messages spawned by these application updaters. Other times I see people completely ignoring vital system messages and error conditions, and we know this because we’ve all done it at some point in time.

Therefore in my mind we need to streamline and simplify the updating of our software. Eliminate any superfluous dialogs and message boxes, and hide the internal complexity from the user. To summarise I believe that all software updates should behave according to these simple rules:

• If the update can be installed without any user intervention then it should be done automatically, with no notification to the user, but allow them to go looking for it specifically.
• If the update needs to interfere with the work the user is doing then it should do as much as it can automatically, and then complete the process when the user has finished with or restarts the application/computer.

Granted these aren’t easy things to implement, but we should do them nonetheless.

So let’s take the Firefox browser as an example of a great application with a poor update process, one that irks me in two main ways. Firstly when it starts it checks if any add-ons need updating, if any have updates it spawns a sequence of dialogs to click through to do the updating. (Does anyone actually skip updating of add-ons?)

Surely this is something that should be handled automatically at start up without the need for any user intervention. Because whenever I start Firefox I just want to visit a website, right now, and clicking/waiting through the dialogs distracts me from that goal

My second Firefox irk is that whenever an update is released and the browser detects it (I always have at least one browser window open), it pops up this large highly-verbose update dialog over whatever site you were visiting at the time. It’s already downloaded and installed it, which is good, but it pops up this annoying dialog prompting you to stop your work and focus all attention on the browser because it needs to restart. However if you select the option to delay updating until the next restart it adds to the annoyance by popping up another useless dialog that you must close before returning to the website that you were on. Why can’t it just prepare the update automatically, and apply it when the browser is next restarted without having to ask the user a senseless question?

Same thing goes for Windows Update, it should install or prepare all of the updates automatically, and silently (this means no popup balloons coming from system tray icons). Then if a restart is required it should just wait until the user decides to shut down or restart the system.

The simple rules described here should eliminate unnecessary interruptions that the user experiences with regards to updating. Fewer interruptions mean that the user can just concentrate on the task at hand, rather than being constantly reminded about some application that they’ve installed. This is a good thing! Distracting the user from their work with your applications updater is wrong and should be banned.

Software developers (this includes everyone involved in design and construction of software – even the managers) need to realise that in the end it comes down to what the user does, rather than what they do it with. Applications are insignificant no matter what you might believe. But judging from the number of applications with crappy updaters out there, there must a lot of people that just can’t or won’t realise this.

Technology should aim to be invisible and blend into our life, rather than be a visible nuisance which we have to tolerate to get stuff done.

So please write better, less annoying, and less visible updaters.