Thoughts and inspirations from a software developer.
I’ve added a page that contains a list of books that I’ve found interesting from both a personal and professional standpoint. More importantly there are links to both Amazon.com (USA) and Amazon.co.uk (UK) in case anyone wants to buy them, of course I get a tiny cut if people do this.
I’ve been doing some Ruby on Rails development recently and one thing that’s bugged me a bit is how to deal with test data while you’re developing the application. Not data that is used just for testing, but data that lives in the development database that you use and fiddle around with while developing the site itself. I’ve found that while developing features of the site I need a fairly large and varied set of data to use, and this data takes a while to enter into the site each time I modify the database.
So instead of repeatedly entering this data I wrote some simple Rake tasks that can automate exporting and importing the data from the database. I can’t take all the credit as the idea was inspired/based on/stolen from the Advanced Rails Recipes book.
Exporting works by writing each table to its own CSV file in the db/dataset directory (or subdirectory) in the rails project. The first line of the file is the name of the columns in the database.
desc "Exports the database contents into CSV files."
task :export => :environment do
require "FasterCSV"
require "FileUtils"
path = File.join(DATASET_PATH, (ENV["DATASET"] || "."))
FileUtils.mkdir_p(path)
ActiveRecord::Base.establish_connection
database = ActiveRecord::Base.connection
(database.tables - SKIP_TABLES).each do |table_name|
FasterCSV.open(File.join(path, "#{table_name}.csv"), "w") do |csv|
# Write column names
csv << database.columns(table_name).map(&:name)
# Write rows
database.select_rows("SELECT * FROM %s" % table_name).each { |row| csv << row }
end
end
end
Importing works much the same way by reading all of the CSV files in the db/dataset directory (or subdirectory) and attempting to load the data into the database. However it does check that the table of that name exists first and skips it otherwise.
desc "Imports data from CSV files into the database."
task :import => :environment do
require "FasterCSV"
path = File.join(DATASET_PATH, (ENV["DATASET"] || "."))
ActiveRecord::Base.establish_connection
database = ActiveRecord::Base.connection
tables = (database.tables - SKIP_TABLES)
database.disable_referential_integrity do
Dir.chdir(path) do
Dir.glob("*.csv") do |file_name|
table_name = file_name[/(.+)\.csv/, 1]
next unless tables.include?(table_name)
rows = FasterCSV.read(file_name)
# Construct a list of column headings from the first row
heading_row = rows.shift.map { |heading| database.quote_column_name(heading) }.join(",")
rows.each do |row|
# Need to escape the data correctly so that inserting is successful
row_data = row.map { |item| database.quote(item) }.join(",")
database.execute("INSERT INTO %s(%s) VALUES (%s);" % [table_name, heading_row, row_data])
end
end
end
end
end
These are faily simple tasks and therefore I’m releasing the code to the public domain, grab the file and use it freely.
To make it work just add it to your lib/tasks directory in your Rails project. The tasks can then be invoked with:
rake dataset:export rake dataset:import
You can also specify a specific subdirectory by passing the DATASET option to rake, like:
rake dataset:export DATASET="dev"
Of course I know there are many things missing from this script, such as being able to specify the rails environment to use, but I’ve kept it simple so adding these features is left as an exercise to the reader.
I’ve been programming in C++/CLI at work recently, and I thought I’d share a snippet/trick or two that I’ve learnt.
When working with the .Net framework in C++ you can come across places where it’s easier to use the Win32 functions to do a particular task rather than the .Net ones (or there’s no direct .Net version of it available). In those cases you may need to get Win32 handles from some .Net classes, or extract primitive types from a handle to an Object instance.
To get the window handle (HWND) of a .Net Form:
HWND form_handle = reinterpret_cast<HWND>(form.Handle.ToPointer());
This also works for other handle types used in Windows, such as HDC.
Effectively this is just unboxing the variable from the .Net type. However the Handle property returns a type of IntPtr and therefore it needs the ToPointer() call to work properly.
Another trick to proper unboxing is when trying to convert from an Object^ to a primitive type, say UINT, or LONG. I discovered that using a safe_cast directly to the primitive type did not work, instead it just raised an exception. So to correctly unbox primitive variables you need to unbox them to a primitive, non-typedef’d type, something like:
int a = 10; Object^ a_obj = gcnew Object(a); // Automatic boxing UINT new_a = safe_cast<int>(a_obj); // Unboxes
Also don’t forget to wrap this unboxing code in a try – catch block as it might still throw exceptions.
It’s funny how things have come full circle in the Windows game development world – when it comes to processing player input of course. It used to be back in the day (I think around 2000 or so) that a game engine would use DirectInput to manage user input, but that has now been deprecated in favour of using plain old native Win32 messages.
(Though you still need to use DirectInput for Joysticks that require it, and XInput for newer devices – such as XBox360 gamepads. There’s an article on how to use both side by side.)
I came to this realisation when working on my own engine recently, as I was planning on coding up a nice primarily DirectInput based system, and only using the native windows functions to handle the mouse input when windowed. However after reading an article or two and looking at some of my old projects I decided against it and just used simple windows message handling to get the job done.
Some of the reasons to no longer used DirectInput are:
Additionally it turns out that not only is using the native windows messages easier to code up, but the code is also shorter and it makes the other architecture cleaner and more elegant to boot.
To use the windows messages as input all you need to do is to process all of the relevant messages:
(There is also another way of getting far more precise mouse values using the more generic input messages, but I did not need this for the project so I did not use it. However developers of first person shooters or other twitch games should investigate this method.)
Then pack the data into your own custom structure so that you don’t have to have the windows message handling values all through your code, and then dispatch the data structure to your game state controller for handling.
Now there are a few things to watch out for when creating this system, mainly dealing with the processing of mouse based input, and especially when operating in windowed mode.
You should cache the mouse position as it is received from the WM_MOUSEMOVE message and only dispatch the new mouse position once per frame in an update function or similar. The reason for this is that the mouse update rate is far higher than the frame rate and movements in the mouse can generate many such messages which flood the system. This is especially bad when expensive computations are performed on the movement of the mouse (such as camera orientation/position) as they will suck up all of the CPU time doing the processing.
When switching the mouse from reporting absolute to relative coordinates there are several things that need to happen. The mouse needs to be captured, the cursor hidden, the current coordinates stored for later, and then set to the centre coordinates of the window. I found that sometimes the mouse cursor show count is greater than one, so I repeatedly call the ShowCursor(FALSE) function to make sure it is hidden. Otherwise the user can still see the mouse as it is wildly moving about the centre of the window. Then at the end when switching back into absolute coordinates don’t forget to restore the previous position of the mouse.
For every mouse that I have encountered the value reported is a multiple of 120 units, with 120 (or WHEEL_DELTA in the code) units representing a single “click” or movement of the mouse wheel. As the Microsoft documentation explains this is so that in future we might have mice that scroll at a finer granularity. Therefore I made sure in the processing that I first accumulated the incoming value and then only reported the movement if it moved more than 120 units.
Additionally I discovered that (at least) in debug mode when I scrolled the mouse wheel I would sometimes get multiples of 120 showing up as the incoming delta. Where as I had not seen this behaviour before at work or otherwise. Therefore it is possible to return more than one “click” or significant movement of the scroll wheel in a single message.
When the user presses down on a mouse button within your client area the mouse should be captured by your application. This is so that you will get a mouse button up message when the button is released while the mouse is no longer above your client window. (That’s also why the WM_CAPTURECHANGED message is listed above.)
One tip for dealing with the keyboard is to only report the first instance of a key down message. You can detect this by checking the status flags on the incoming message (Bit 30 of the wparam parameters I believe, but check the documentation). This blocks the key repeat feature of the messages from going to the rest of your application so you’ll only get one key down event. Hence enable it for the general game but disable it when the user is typing into a text input field, as you’ll want to allow the default system key repeat settings to take effect.
That’s it for now. Hopefully this has helped someone that was in a similar situation to me before.
This is an update on the game development project I am working on as a hobby. Initially the goal was to make an engine and a small game an enter it into the TSumea AI programming challenge. However I wasn’t able to complete the entry in time as I really bit off far more than I could chew.
It came down to the fact that developing an engine, a game, and an interesting AI simulation is far more for one person to do in three weeks of hard coding. There are some questions to consider as to why I didn’t have an entry ready.
Regardless, the final result is that I didn’t get the engine or game completed. Not only that, but from all the people that suggested interest in the challenge there was only one final entry. How disappointing.
But still I have developed something that looks like an engine.
In three weeks “working” about 12 to 16 hours a day of programming/research I have managed to create a nice engine, with (what I hope is) a good architecture underneath it. I even spent a weekend working out mathematical and physics formulae (for collision detection and response) on my Tablet PC, and I can honestly say that I haven’t done that much mathematics since University.
(I have to note that it’s quite nice to scribble out mathematics on a tablet. As you can use normal handwriting with the correct mathematical symbols and all on a grid ruled background. Also you can move stuff around and erase with ease if you make a mistake.)
For the time spent on the project I have had a blast. It reminded me of the good old Uni days staying up till the early hours in the morning coding, and yes I did that this time as well. I learnt to use some interesting bits of the Boost library, experimented with better ways of writing game architecture code, and re-learnt a whole bunch of mathematics and phyics.
However the engine and game is not complete, and since I stopped working on it on the 15th I haven’t done much to it since. (I’ve been playing games instead!) But I’ll start working on it again now and see if I can create a decent game out of it. Also I have a couple of articles that are in various stages or preparation and will post them when they are done.
Have a happy new year too!
Now that I’ve actually been using Visual Studio for a lot of development work at home I’ve encountered some annoyances. One of these has been that sometimes when switching between windows using the Ctrl+Tab/Ctrl+Shift+Tab shortcuts it gets stuck and refuses to take any more keyboard input until I click on it with the mouse.
A quick glance at the problem reveals that it’s a focus issue with the window that pops up listing the open documents, where if you switch quickly enough it doesn’t get time to properly return focus to the main window.
Luckily there’s a handy solution out there that fixes the problem, you simply swap what the Ctrl+Tab and Ctrl+F6 keyboard shortcuts are bound to (but don’t forget the Shift variants).
I get really annoyed by obtrusive software update mechanisms. They steal focus, popup dialogs, show message boxes, add random icons to your system try, and then popup balloon items. All in a vain attempt to let you know that some crappy little app that you rarely use has an update available and that you should stop and drop everything that you are doing to install the update right now because it really can’t wait one second longer! Obviously someone involved making the software thought that keeping the little application up to date is a far better use of your time than you doing the things you need/want to do.
The main gripe that I have with software updaters is that they announce the update at completely arbitrary and inopportune times, often covering whatever you were actually trying to do with the program. Or better yet you’re concentrating and getting stuff done and up comes a window asking you to restart the application/computer. And those are just the friendly updaters, some just restart the application right smack bang from under you, destroying whatever you were working on. There are many offenders that I could name, but just off the top of my head I can call out Java, Acrobat Reader, Logitech software, Paint.Net, Apple/iTunes, and Firefox as programs with bad update mechanisms. (Some worse than others of course!)
A particularly special gripe I have for the Windows updating mechanism. The updates do warn you if the computer will need to be restarted after installation, but almost every single update states that it may need to restart my system. So it’s pretty much random as to which updates will force your system to restart and which one’s won’t. The other problem is that the options for how the update system works are all undesirable in my view.
I know of many instances where people have been either confused or panicked by simple update messages spawned by these application updaters. Other times I see people completely ignoring vital system messages and error conditions, and we know this because we’ve all done it at some point in time.
Therefore in my mind we need to streamline and simplify the updating of our software. Eliminate any superfluous dialogs and message boxes, and hide the internal complexity from the user. To summarise I believe that all software updates should behave according to these simple rules:
Granted these aren’t easy things to implement, but we should do them nonetheless.
So let’s take the Firefox browser as an example of a great application with a poor update process, one that irks me in two main ways. Firstly when it starts it checks if any add-ons need updating, if any have updates it spawns a sequence of dialogs to click through to do the updating. (Does anyone actually skip updating of add-ons?)
Surely this is something that should be handled automatically at start up without the need for any user intervention. Because whenever I start Firefox I just want to visit a website, right now, and clicking/waiting through the dialogs distracts me from that goal
My second Firefox irk is that whenever an update is released and the browser detects it (I always have at least one browser window open), it pops up this large highly-verbose update dialog over whatever site you were visiting at the time. It’s already downloaded and installed it, which is good, but it pops up this annoying dialog prompting you to stop your work and focus all attention on the browser because it needs to restart. However if you select the option to delay updating until the next restart it adds to the annoyance by popping up another useless dialog that you must close before returning to the website that you were on. Why can’t it just prepare the update automatically, and apply it when the browser is next restarted without having to ask the user a senseless question?
Same thing goes for Windows Update, it should install or prepare all of the updates automatically, and silently (this means no popup balloons coming from system tray icons). Then if a restart is required it should just wait until the user decides to shut down or restart the system.
The simple rules described here should eliminate unnecessary interruptions that the user experiences with regards to updating. Fewer interruptions mean that the user can just concentrate on the task at hand, rather than being constantly reminded about some application that they’ve installed. This is a good thing! Distracting the user from their work with your applications updater is wrong and should be banned.
Software developers (this includes everyone involved in design and construction of software – even the managers) need to realise that in the end it comes down to what the user does, rather than what they do it with. Applications are insignificant no matter what you might believe. But judging from the number of applications with crappy updaters out there, there must a lot of people that just can’t or won’t realise this.
Technology should aim to be invisible and blend into our life, rather than be a visible nuisance which we have to tolerate to get stuff done.
So please write better, less annoying, and less visible updaters.
Since I’ve got some free time now I’ve decided to write a nice and simple 3D engine, again. The main reason for this is to enter a small programming competition, but I’m taking care with the engine design and architecture so that I may use it in the future. Also it will be a nice addition to my portfolio.
My last attempt at making a game engine was over 7 years ago, back when I was learning all of this programming and mathematics stuff. Looking back over the code now I shudder at all of the simple and naive mistakes I made, and the faulty assumptions I was working with. Granted at the time the engine did have some spiffy features like a scriptable Quake3-like shader engine among other things. But in the same vein I made some serious snafu’s, like building my own container classes that could actually return null references! Though overall the learning experience of writing such a large amount of code for a project was the best thing that came out of it. (I’ll post the source code sometime so that people can learn from my mistakes.)
However this time around I’m putting in my years of experience and building this engine right. I’m using existing libraries whenever possible (go STL, ATL/WTL, and other acronyms) and thinking about what I actually need and implementing only that rather than spending a lot of time developing features I won’t use. Hopefully this will mean a reduced development time and a better finished product.
I’m going to update this blog on my progress and post some demos and screen shots of the project as it’s progressing.
Just found a handy list of development blogs. It appears I have most of the one’s at the top of the list on my RSS reading list, so I’ll be checking some of the others out as well.
Then again I’ve been spending a fair bit of time on StackOverflow.
Initially I wrote this rant some time ago after reading a bunch of IT related news stories that just irritated me greatly. Enough that I actually had to vent and writing it down was quite soothing. I present this as a post now after editing it and moderating the wording to make it sound less angry (because as we all know we can say some pretty stupid things when we let the anger speak). So here it goes…
I get annoyed constantly when I see or hear an article or advertisement where people lump everybody that work in the computing field into the “IT worker” category. It’s irritating when journalists do it, but it really riles me up when I see the ads on TV for those “IT schools” that claim you can have a career in IT by doing a simple six week course. It’s annoying because the computing field is very broad and applying the IT label to everything just cheapens it.
To me applying the IT label to the field of computing is like lumping electricians, electrical engineers, and high energy physicists in the same group.
They’re not!
Electricians are tradesmen that work with their hands performing practical tasks like building or repairing existing electrical devices and wiring. Electrical Engineers design and create entire new sets of devices from ideas and available components. While the high energy Physicists pound atomic particles together creating quarks, neutrinos, and tiny black holes. Each occupation performs an essential task in society, but each occupation requires a completely different skill and mind set. Therefore it makes no sense to lump them into such a broad category and assume the skills they have are interchangeable.
Would you ask your particle physicist friend to install your home wiring? Or ask an electrician to design a new television? I’d wager not.
To put this into the computing perspective, IT people are the one’s that maintain the computer systems that we use to do our work every day (a thankless job no doubt). Software Engineers design, develop, and test new pieces of software, creating the programs people use from nothing but mere ideas from one’s head. Computer Scientists on the other hand get involved with research related to how we use computing resources. As you can see these are quite varied tasks, each of which requires a different set of skills and a different mentality.
So naturally I get annoyed when people classify all computer based occupations such as Software Engineering and Computer Science in with IT.
I have an idea that this is partly due to the image that has been cultivated by society of anyone that works with computers being a stereotypical geek/nerd working in a dark and dingy office. Over time this image has been self-reinforcing and has created a barrier between the “computer geeks” and the rest of society, especially in business. It is this barrier that has made people simply lump all computing professions into the IT category.
For this situation to change we need to change the culture in the industry, then we need to raise awareness of the wide spectrum of available professions within computing. Once this happens people will realise that there is more to computing than just system administration, or just programming, or just playing games, and then they might stop using the IT moniker.
Therefore for now I urge people to not automatically group all of the computing fields as being the IT industry, there’s far more to it than just computer management.
And yes, this means that I will not fix your computer!
