RedGem: a 3D game engine scripted in Ruby

I wanted to see how far Ruby could actually go as the scripting language for a 3D game engine, so I wrote one. It's called RedGem. It's got an editor, physics, a renderer, and all the gameplay logic is pure Ruby.

This post is a quick breakdown of how it's put together, focusing mostly on where Ruby fits into the architecture.

Quick disclaimer: it was built with heavy help from AI. It's also experimental, and it isn't built from zero: it glues together a handful of great open source projects, listed below.

The idea

In RedGem, a game is just a folder. Levels are glTF files made in the editor or exported from Blender, and gameplay is defined by a bunch of Ruby files. A typical behaviour looks like this:

class Player < Behaviour
  SPEED = 6.0
  JUMP = 9.0

  def update(dt)
    move = Vector3.new(Input.axis(:horizontal), 0, -Input.axis(:vertical))
    character.move(move.normalized * SPEED)
    character.jump(JUMP) if Input.action_pressed?(:jump) && character.grounded?
  end

  def on_trigger_enter(other)
    other.destroy if other.tag?(:coin)
  end
end

You attach it to an object, press Play, and it runs.

The layers

The engine itself is written in C++17 and doesn't reinvent much: Jolt handles physics, raylib creates the window and talks to OpenGL, cgltf parses levels, Dear ImGui renders the editor UI, and Blender is used for modeling. Ruby is mruby, compiled right into the binary.

The build produces a few small programs instead of one big one: the editor, a player that only knows how to run a game (a built game is a copy of it, about 7 MB), a relay for multiplayer, and a test runner. So the game your players get carries no editor and no test code.

The Ruby layer splits into two parts.

Native bindings are basically C++ functions exposed as Ruby methods like transform.position=, Input.axis, and Physics.raycast. They're intentionally kept dumb. A Ruby GameObject just wraps an integer ID, and every native call uses that ID to look up the real object in the C++ scene graph. No pointers cross over to Ruby, meaning if a script hangs onto a dead object, it throws a clean Ruby exception instead of a crash.

The prelude is standard Ruby bundled straight into the binary. Behaviour, Vector3, SceneScript, timers and even the title and pause menus are all written in Ruby on top of the raw bindings. Keeping anything that didn't strictly need C++ out of C++ made iterating on the API super cheap.

One frame

Every call from C++ into Ruby is wrapped and protected. If an update method throws, the engine catches it, logs the backtrace, disables that specific behaviour, and keeps the game running. When you're actively tweaking a game, that resilience matters way more than raw execution speed.

Ruby things I ended up using

A few specific Ruby features turned out to be super useful here.

const_missing handles script loading. Whenever a script references Patrol for the first time, the engine automatically checks for scripts/patrol.rb and pulls it in. No one has to write require.

Keyword arguments pass down configuration. Doing enemy.add_component(Patrol, speed: 4.0) attaches the component, and inside it, prop(:speed, 1.5) reads that value, falling back to a property set in Blender, then to the default.

Fibers drive the test suite, which lives in its own tool (redgem-test). A system test is just a Ruby block simulating gameplay, where each wait yields the Fiber back to the engine for a single frame. This lets tests read linearly from top to bottom even though they span hundreds of frames:

SystemTest.define do |t|
  t.hold(:w, frames: 60)
  t.assert("the coin was collected") { Scene.find("Coin1").nil? }
end

Blocks make timers read like plain English. Every behaviour gets after, every and tween, and they're a small piece of plain Ruby in the prelude, with no C++ involved:

after(2.0) { destroy }
every(0.5) { Scene.spawn("Bullet", position: muzzle) }
tween(0.4) { |t| transform.scale = Vector3.one * (1.0 + t) }

Levels can also be set up entirely via Ruby code. A single class maps behaviours to objects, keeping the level file clean so it only tracks what exists and where.

What a game gets

Games take keyboard, mouse and gamepad input, draw menus and HUDs, play sounds, throw particles, blend animations per character, save progress between runs, and build for Linux, Windows and Android (with touch controls that show up on their own).

They can also be played together over the network. One machine hosts, the others join, and scripts talk in named messages:

Network.on(:hit) { |data, from| @health -= data[:damage] }
Network.emit(:hit, { damage: 3 })

Players on the same Wi-Fi find each other's games automatically. Over the internet they meet through a small relay that ships with the engine: one opens a room, gets a code like K7QM3P, and the others type it in.

Scripts are sandboxed, too: the embedded Ruby has no files, sockets or eval, so a game can't touch the player's machine.

What it cannot do yet

The editor is developed on Linux (there's a Windows build, but it has only been tried under Wine), Android games have only run on an emulator, nobody runs a public relay for you, and there are no blend trees or lit particles. It's a young, experimental engine, and the repo keeps an honest list of known limitations: a good place to start if you want to contribute.

There's a short demo, ready-to-run downloads and a bunch more screenshots up on the project page. The source is on GitHub, MIT licensed.