Moving On

26 September 2014

It's been over 7 years now since I've moved to Montréal, and it's been a fairly eventful 7 years. I've done all sorts of cool projects like Pornhub (warning: adult content), various startups, and day-trading to name a few. I've met many extremely talented people here in a city with a huge array of backgrounds: technical, business, artistic, musical to just name a few. The city that blends North America and Europe, while throwing in a mix of Québecois culture - walk down Ste. Catherine street when the Habs beat the Bruins to experience some of this. The people here seem genuinely interested in more than just careers or self-progression; they love art, culture, or just plain old having a good time.

For all the great things going for this place, I am not someone who likes to sit still for an extended period of time. Therefore I am noting here with a slight regret that I am leaving Montreal this November for a new adventure. I've stayed this long to help some people I respect build a startup called sweetiQ; however I've reached a point where I can be satisfied with having made a significant contribution, and that they will be able to carry things forward without my help.

Assuming that the visa and background checks go through, I'll be heading to San Francisco to work for Youtube. This is a completely new experience for me, having never worked for a large company before; heck I've never even worked for an established team - even for Pornhub it was pretty much a brand new project and every one of us had the freedom to decide how things were going to get done.

As part of this new adventure, I'm hoping to fall back into blogging more often. I was fairly active (2-3 times per week) back in 2008-10 and I enjoyed it, so I'm hoping to resume that habit with some of the new adventures I'll be having; one of which will be describing the shift from small startups to a big company. Stay tuned, as there will be more to come.

By Rob Britton

World Generation Project

31 August 2014

Inspiration

I've started up a project recently to explore all sorts of different types of procedurally generated content - terrain generation, weather, economics, maybe even cool stuff like culture. I've been hugely inspired by Dwarf Fortress in this regard; you fire it up and it generates a hugely complex world for you to play around in. I'm not a big fan of level of micromanagement required to play the game itself and so I haven't gotten too deep into it, but reading into the articles of how they are building it is extremely interesting.

In addition to Dwarf Fortress, I've been playing games like Terraria and Starbound (for some reason I never got into Minecraft) which also feature a fair amount of procedural generation. I was especially impressed with Starbound which unlike the others builds an entire universe of planets to explore rather than just one.

Upsides

Using these types of technologies to generate worlds, you can have near-limitless new content for someone to explore (I have no idea how big the universe is in Starbound). Once you "beat" the game, you can just restart in a new world to get a brand new experience - the worlds and structures within those worlds will change completely.

Because of the level of content available through these methods, it makes a major impact from a business perspective. If you don't need to hire loads of people to generate and maintain content for the game, you can launch a business with much less startup capital and maintain a much lower burn rate than you would otherwise. This in turn allows these types of companies to focus on more niche audiences, which arguably makes the games more interesting for those people.

Downsides

The problem that I've ended up finding with these games is that once you play it a bit, it gets fairly monotonous. The worlds do indeed change from game-to-game and while technically each world is different, the worlds do not differ in any significant way to be interesting. For example in Starbound you can go to a variety of different types of planets and see different cities and dungeons for different races, but once you've played the game and explored these a little bit, they are more or less the same each time.

The Plan

What I intend to research with this project is exploring more complex procedural generation that can potentially construct a truly unique experience each time you play it. Part of this research is based off of what Dwarf Fortress is focusing on: simulation of civilizations, politics, and economics. I believe that with these additional aspects, a system can be built that generates completely unique and interesting worlds.

Initially I do want to spend some time on terrain generation, since that can have major impacts on the development of economies and politics of a society. It would be interesting then to see how changes in the terrain generation process impact these. Over the next little while I'll be posting a bunch of articles about terrain generation - be prepared, some of them may include a bit of math.

Some Code

You can see the code that I am using for this in my open-source Ruby library Worldgen.

By Rob Britton

LED Grow Lamp Update

22 June 2014

Some people have asked me how my Grow Lamp project worked out. To recap, it was a simple little board with a bunch of different coloured LEDs hooked up to a battery that would try to get a plant to grow without any sunlight.

To be quick, the project failed. The two main problems turned out to be:

  • Sunlight still overpowered the grow lamp. The plants ended up still being more sensitive to the sunlight (even in the not-so-well-lit room) so they would just reach towards the sunlight and more or less “ignore” the grow lamp.
  • Power consumption was too high. I was using a 9V battery to power the apparatus, and the batteries were dying out in usually a day or so. This is not a long-term sustainable solution since I’m not terribly interested in constantly buying and disposing of 9V batteries.

Both these problems are fixable. For the sunlight issue, I can attempt to grow the plants inside a dark closet (like where I brew wine). This will let me see exactly how the lamp is doing. For the issue with the power consumption, I can probably hook up some sort of power supply up to the main power to the house rather than relying on batteries.

The main reason why I haven’t really tried either at the moment is simply time – I haven’t really gotten around to doing either.

By Rob Britton

MongoDB and $exists Queries

12 March 2014

Since Mongo is a schema-less database, you’re not guaranteed to have the same field in all your documents. Often you’ll find yourself querying on a collection to only find documents containing that field:

db.myCollection.find({myField: {$exists: true}})

As it turns out, the $exists query does not use any indexes provided. If you’re working on a large dataset, this means that doing this type of query can take quite a long time.

One solution that we’ve found that works great and is much much faster is to use a sparse index on the field, and then query on a degenerate regex:

db.myCollection.ensureIndex({myField: 1}, {sparse: true})
db.myCollection.find({myField: /.*/})

Obviously this won’t work if myField is not a string, but there are definitely other ways to avoid the $exists query that does not use an index.

By Rob Britton

Why We Ditched Socket.IO

10 December 2013

Last summer when building SweetiQ, we had a need to use WebSockets for real-time communication with the client. The obvious choice was to use socket.io, which is the library that everybody uses since it simplifies a lot of WebSocket details, and allows it to work on browsers that don’t yet support it (this was more important a year ago than it is today, but meh).

I’m going to tell you why it was a bad choice.

Our first major issue is that when clients left the browser open for a long time, the socket.io connection would die. Not the end of the world, since socket.io has a reconnect capability. Unfortunately we had to do a bit of logic on socket reconnects, and the documented “reconnect_failed” event was never firing. I discovered this issue on Github which said that other people were having the same problem, so it wasn’t just me. I can’t remember where I saw the fix for it, but I posted a tiny code change on the issue so that other people are able to use the event properly. I can’t remember at this point whether there was a pull request or not, or whether I saw the fix on Stack Overflow; whatever the case is I didn’t make my own pull request.

People continued to have the same problem, and many people commented on the issue saying that the little fix I posted works. There are now three outstanding pull requests that supposedly fix the issue, yet none of them have been merged. It has now been over two years since this issue was opened, and the two line fix is still not in the core socket.io.

We started seeing some other issues: socket.io couldn’t get through firewalls, certain antivirus software would block the connection, etc. I’m not 100% certain what the issue was since the clients we had were on large corporate networks somewhere out there on the Internet.

One of my coworkers discovered SockJS, which is a library that provides a WebSocket-like system that uses the exact same interface as a native WebSocket (onopen, onclose, onmessage) and has a server-side supported in many different languages – which was handy, since we periodically have issues with Node.js and less-mature libraries than the other language we use daily (Python) so we had an option in the case we wanted to switch. Since we had no real idea on how to solve the corporate firewall problem, we ended up giving SockJS a go. The migration was pretty simple since the way you write code with SockJS and the way you write code with socket.io aren’t that different.

It worked beautifully. Every single one of our clients were able to use the WebSocket-based features without any problems at all, including those behind corporate firewalls. To this date we have not found a single bug with it, and it continues to work great a year into production without any updates.

I would recommend that if you’re using a web app that uses any sort of WebSocket functionality, I would recommend using SockJS. If you’re not using any sort of library then you’re even better off, as it is a drop-in replacement since everything works exactly the same as the normal WebSocket specification.

By Rob Britton

See the index for more articles.