Lua dofile and globals


I have been doing dissector development for Wireshark in Lua for a while now. Lua’s mechanism for reusing code organized across multiple files comes in two flavors, dofile and require. For historical reasons, we’ve been using dofile quite exclusively.

Our dissector files mostly contain global functions, and local or global tables. The tables are used to determine which function should be called to dissect a particular message, to look-up names that map to particular key values, and so on. If a script file needs a function or table in another file, we simply dofile the other file at the top. Lua interpreter will recognize all globals (functions and variables without local) of the other file from that point onwards.

Let’s consider an example. Here’s file1.lua

function f1()
  print("f1 called")
  f2()
end

dofile("file2.lua")

f1()

Here’s file2.lua, located in the same folder as file1.lua

function f2()
  print("f2 called")
end

When you run file1.lua, this is what you get

$ lua file1.lua
f1 called
f2 called

Lua only encounters call to global f2 inside function f1 when we call function f1 towards the end of file1.lua. If we call f1 right at the beginning of file1.lua we get

$ lua file1.lua
lua: file1.lua:2: attempt to call global 'f1' (a nil value)
stack traceback:
	file1.lua:2: in main chunk
	[C]: in ?

If we move dofile towards the end of file1.lua we get

$ lua file1.lua
f1 called
lua: file1.lua:3: attempt to call global 'f2' (a nil value)
stack traceback:
	file1.lua:3: in function 'f1'
	file1.lua:7: in main chunk
	[C]: in ?

Summing it up, a global exists only when Lua has encountered it, whether within the same or another script file.

Let us explore one other characteristic of dofile with another example. This is file1.lua

dofile("file2.lua")
hello['world'] = 'hello!'
print(hello['world'])

function f1()
  print(hello['world'])
  f3()
end

dofile("file3.lua")

f1()

It requires a certain global table in file2.lua, which it updates before using. It also requires a certain function f3 in file3.lua. This is file2.lua

hello = {['hello']='world!'}

function f2()
  print("f2 called")
end

This is file3.lua

dofile('file2.lua')

function f3()
  print('f3 called')
  f2()
end

It needs function f2 in file2.lua. You shouldn’t need dofile because file1.lua has already loaded file2.lua, but whoever coded file3.lua probably doesn’t know that. Let’s execute file1.lua and see what happens

$ lua file1.lua
hello!
nil
f3 called
f2 called

By the time function f1 gets called, the value of key 'world' in the hello table ceases to exist. Why? Because, right before f1 is called, when we dofile file3.lua it will dofile file2.lua. Since file2.lua gets interpreted again, the global hello is replaced by a new table.

Summing it up, Lua will load and execute the same file again when it encounters dofile, redefining all globals in it.

That is why, for reusing code in other script files, use require. Lua will not reinterpret a script file that has already been encountered before. Modifying the example scripts above is straightforward. Replace all dofile with require and drop the lua extension. Thus dofile('file2.lua') will become require('file2'). Execute file1.lua thus

$ export LUA_PATH=$PWD/?.lua
$ lua file1.lua
hello!
hello!
f3 called
f2 called

Globals now work without side-effects! Note the use of LUA_PATH environment variable to specify the search path for Lua scripts. You can also set search path inside a Lua script by modifying package.path.

What if you have already invested in dofile and don’t want to change things for now? In the example above, modifying file2.lua thus, eliminates our problem

if hello == nil then hello = { } end
hello['hello']='world!'

function f2()
  print("f2 called")
end

Wireshark migrating to Qt


Wireshark on Mac OS X is downright ugly. It requires XQuartz, an X11 port for OS X. Wireshark have announced that they have begun a steady migration to Qt. The move has its critics, and backers. I back whatever looks better than it currently does. As Miguel de Icaza of Xamarin points out, it is possible to look good on Mac OS X with the Gtk+ port for Cocoa. So what gives?

Handling TCP Keepalive


TCP Keepalives are useful for scenarios where one end of the connection disappears without closing the connection. This can happen when a NAT or firewall resets, or forcibly closes the connection. The following code in Python enables sending a Keepalive message when there is no data activity over a socket for 60 seconds. If it does not get a response, it sends Keepalive messages 4 times at intervals of 15 seconds. After that, the connection is closed. The code has been tested with Python 2.7.2 and Ubuntu 12.04.

To test, create a TCP listener/server using netcat on a different PC (or nc on a Mac)

netcat -l 192.168.0.120 8001

Now, disable the network connection, enable a firewall, or power off the router, to see Keepalive in action. Wireshark highlights Keepalive messages when TCP sequence number analysis is enabled.

You’ll see the following messages when the connection times out

Traceback (most recent call last):
  File "socket_test.py", line 39, in do_work
    req = sock.recv(10)
error: [Errno 110] Connection timed out
Other Socket err, exit and try creating socket again

On Mac and Windows you can enable Keepalive, but cannot set TCP_KEEPIDLE and other parameters, you’ll get the following error message (Python 2.7.2 with macports)

Traceback (most recent call last):
  File "socket_test.py", line 65, in <module>
    do_work()
  File "socket_test.py", line 19, in do_work
    sock.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, 60)
AttributeError: 'module' object has no attribute 'TCP_KEEPIDLE'

File virtualization in Windows


Windows versions since Vista have a feature in which files with administrative privileges in “Program Files” and other folders, may be masked by a version of the file in user’s Virtual Store. The Virtual Store is located at C:\Users\User_name\AppData\Local\VirtualStore.

If you have changed a file under Program Files using administrative privileges, but find the changes mysteriously fail to have any effect, the Virtual Store may be to blame. I noticed this when editing Wireshark’s init.lua file, but the changes would have no effect. I was able to fix the problem by editing the file stored at the Virtual Store location mentioned above. Certain editors are aware of this Windows feature, when opening and saving a privileged file they use the Virtual Store without informing the user.

Dealing with segmented data in a Wireshark dissector written in Lua


Protocols based on stream-oriented transport protocols like TCP may get segmented i.e. the PDU boundaries may not be preserved. If you are not familiar with writing dissectors for Wireshark in Lua, read Create a Wireshark dissector in Lua.

Protocols based on TCP

You can simply let the TCP dissector reassemble segments by telling it how much data the dissector function still needs.

Adding the following logic to the dissector function does the trick. Reading the PDU length field is protocol specific, change it appropriately.

    local i = 0
    repeat
        if buf:len() - i >= 2 then    -- change length field size appropriately
            -- we have length field
            i = i + buf(i,2):uint() -- change appropriately
            if i > buf:len() then
                -- we don't have all the data we need yet
                pkt.desegment_len = i - buf:len() 
                return
            end
        else
            -- we don't have all of length field yet
            pkt.desegment_len = DESEGMENT_ONE_MORE_SEGMENT 
            return
        end
    until i >= buf:len()
    
    -- rest of the dissector function remains the same,
    -- but use a repeat / until loop to read all PDUs

The TCP dissector calls the dissector function with progressively larger chunks of data taken by aggregating data from the following TCP segments. Once data has been aggregated, your dissector function will proceed as usual.

Protocols not based on TCP

If your protocol is not based on TCP, you will have to handle segmentation on your own. It bears to keep in mind that Wireshark will call the dissector function for each packet, from first to the last, when you first open a capture file. This knowledge can be used to preprocess each packet, and save some state information such as whether it is segmented or not. If it is segmented, its raw data can be appended to a global byte array. This process will repeat until the whole message is ascertained to have been read. A new Tvb can then be created and used for dissection. The final byte array is saved in state of the last packet in sequence.

    local state = pktState[pkt.number]

    if state ~= nil then
        -- we've already processed this packet
        if state.complete == true then
            pkt.info = "Command [complete]"
            buf = ByteArray.tvb(state.buffer, "Complete Command")
        else
            pkt.info = "Command [incomplete]"
            return -- nothing to do
        end
    else
        -- first time here, capture file has just been opened?
        state = {}
        if partialBuffer == nil then
            partialBuffer = buf(0):bytes()
        else
            partialBuffer:append(buf(0):bytes())
            buf = ByteArray.tvb(partialBuffer, "Command") -- create new tvb for packet
        end
        local i = 0
        repeat
            if buf:len() - i >= 2 then -- change length field size appropriately
                -- we have length field
                i = i + buf(i,2):uint() -- change appropriately
                if i > buf:len() then
                    -- we don't have all the data we need yet
                    state.complete = false 
                    pktState[pkt.number] = state
                    return
                end
            else
                -- we don't have all of length field yet
                state.complete = false 
                pktState[pkt.number] = state
                return
            end
        until i >= buf:len()
        state.complete = true
        state.buffer = partialBuffer
        pktState[pkt.number] = state
        partialBuffer = nil
    end

    -- perform dissection of buf

Remember to initialize pktState to an empty table i.e. {} in the init function. partialBuffer is a file local variable used within the dissector function.

Logging to Wireshark’s Lua console


A Lua script can log to the Lua Console of Wireshark by using inbuilt utility functions. For instance warn(“hello world”), prints that text to the console prefixed with date and time information and the text WARN. The console needs to be open for the information to appear. It can be opened from Lua sub-menu under Tools.

Wireshark and Lua bitwise operations


Lua has shipped a bitwise library since version 5.2. Wireshark Lua implementation has had the same bitwise operations, but the functions are accessed using table name bit instead of bit32.

If you are not familiar with writing dissectors for Wireshark in Lua I recommend reading this post first.