Please take a few minutes to complete this short survey on service testing.

Scripting Go: Embedding Lua in Your Go Apps

  • 2017-03-24

When video games are written, the game engine would be written in some low level language such as C or C++ to achieve the best performance. You would need to get direct access to the hardware and such languages are perfect for that. Building software in such languages requires good understanding of OS, memory management and just internals in general. To make game development more accessible, for instance to UI or level designers, we can introduce a scripting language that can hook into the core engine and perform some actions as the game state changes. The idea of scripting your core is amazing.

We can see similar uses in other areas. NGINX web server has support for Lua scripting which allows you to hook into request processing logic. From there you can filter or log requests based on some custom rules. Varnish HTTP cache has some similar functionality which lets you decide how to cache your content. My friend at my old job scripted report generation logic in a C service, which allowed us to get arbitrary aggregations from raw data available in the memory. There are plenty of use cases for scripting the core of your technology.

In this article I want to take a look at how we can embed Lua language into Go applications.

The Problem

Let's say we want to build a simple event filtering application. The application will do the following:

  1. Read event from some source.
  2. Apply event validation code to the event.
  3. Forward event if validation passes, or drop the event if it does not pass the validator.

Enter Lua Filter

For the sake of simplicity, I'm going to use stdin for the event source and I will forward valid messages to stdout. We can assume one event per line.

Let's start by defining our filter.

package main

import (
    "bufio"
    "errors"
    "github.com/yuin/gopher-lua"
    "os"
)

type Filter struct {
    state                    *lua.LState
    exceptionHandlerFunction *lua.LFunction
}

There are several Lua implementations for Go. I decided to go with gopher-lua since it's a pure Go implementation. The filter is a wrapper around Lua state object. We'll be adding helper methods to our filter that operate on the state.

func NewFilter() *Filter {
    state := lua.NewState()

    filter := &Filter{
        state: state,
    }
    filter.exceptionHandlerFunction = state.NewFunction(
        filter.exceptionHandler)
    return filter
}

func (f *Filter) exceptionHandler(L *lua.LState) int {
    panic("exception in lua code")
    return 0
}

To instantiate a new filter we'll call NewFilter. The exception handler function can be used to capture exceptions coming from the lua code. For example, when we get an exception we could investigate lua stack and send the data somewhere. I'll leave that as an exercise to you.

func (f *Filter) LoadScript(filename string) error {
    return f.state.DoFile(filename)
}

Loading Lua scripts is easy. gopher-lua comes with a helper DoFile which reads the file into the state. When we load the script, it's actually going to be executed. All statements will be executed. If you have function definitions, those functions of course will not be auto-called, but their symbols will be added to the state.

The next step is to perform a small validation.

func (f *Filter) ValidateScript() error {
    fn := f.state.GetGlobal("filter")
    if fn.Type() != lua.LTFunction {
        return errors.New("Function 'filter' not found")
    }
    return nil
}

After we load the script, we want to do some simple sanity checking and make sure that we have filter function available. Later when we read our event we are going to be calling this function and passing it in.

Here's the event validation method.

func (f *Filter) ValidateEvent(event string) (bool, error) {
    fn := f.state.GetGlobal("filter")

    f.state.Push(fn.(*lua.LFunction))
    f.state.Push(lua.LString(event))

    // one argument and one return value
    err := f.state.PCall(1, 1, f.exceptionHandlerFunction)
    if err != nil {
        return false, err
    }

    top := f.state.GetTop()
    returnValue := f.state.Get(top)
    if returnValue.Type() != lua.LTBool {
        return false, errors.New("Invalid return value")
    }

    return lua.LVAsBool(returnValue), err
}

It takes a string. We're getting reference to our filter function and we're prepping the lua state for a call. Lua VM is register-based, and it looks very similar to real CPU design. If you were in x86 assembly, in order to call a function you'd push arguments onto the stack and issue call. Similarly, in lua we are pushing out function pointer onto the stack and the arguments to the function onto the stack. To invoke the function we call PCall. This method takes the number of arguments the function takes and number of return values. We're also passing in our exception handler to catch errors.

When a function call returns, the values returned by the function will be at the top of the stack.

The code that glues all of this together is our entry point to the application.

func main() {
    if len(os.Args) != 2 {
        println("provide filter script")
        return
    }

    filter := NewFilter()
    err := filter.LoadScript(os.Args[1])
    if err != nil {
        panic(err.Error())
    }

    err = filter.ValidateScript()
    if err != nil {
        panic(err.Error())
    }

    scanner := bufio.NewScanner(os.Stdin)
    for scanner.Scan() {
        event := scanner.Text()
        isValid, err := filter.ValidateEvent(event)
        if err != nil {
            panic(err.Error())
        }

        if isValid {
            println(event)
        }
    }
}

This code is very simple. Let's write our filter function.

function filter(line)
    found = string.find(line, "if")
    if found == nil then
        return false
    end

    return true
end

The filter is very simple -- we're just filtering out lines that do not contain substring if. Let's run the filter on our source code.

$ cat lua-filter.go | ./scripting-go filter.lua
    if fn.Type() != lua.LTFunction {
    if err != nil {
    if returnValue.Type() != lua.LTBool {
    if len(os.Args) != 2 {
    if err != nil {
    if err != nil {
        if err != nil {
        if isValid {

This is great. We can modify filter.lua script without having to recompile entire application. There are some weaknesses of course. This is not a true lua 'jail' meaning that a rogue lua script could halt execution of our app. That's another topic.

Entire source code is available on github.