Exchanging Values and Calling Functions

The Stack (and using values directly)

In the native APIs of Lua and Squirrel, you use a virtual stack to exchange values between native code and script code, and to perform most (if not all) operations on objects. This is mostly because these language implementations have their own garbage collectors which have to keep track of which objects are in use. Using a stack is a fairly elegant solution to this problem, as any objects which the native code is manipulating are kept on the stack. (In contrast, some language implementations, like Ruby, use non-portable solutions such as scanning the C stack for GC-held objects.)

With MiniD, this is not a problem, because the MiniD garbage collector is the D garbage collector. This means that you can manipulate values with the native API much more directly. This also obviates the need for other mechanisms necessary for keeping track of garbage collected objects, such as reference counting, and the Lua "registry" and "reference" systems.

However, MiniD still uses a stack, although for fewer tasks than the other languages. Basically it's only used for passing parameters to functions, and getting return values from functions. After getting hold of the values off of the stack, you deal with the values directly (or nearly directly, since many operations are done as methods of a state anyway).

Pushing and Popping and Just Plain Retrieving

The stack is held in an MDState instance. This class represents a thread of execution. This includes a call stack and a value stack which holds all the local variables (and any values which have been passed to native functions). All pushes and pops are done through an MDState instance.

The two main methods are -- wait for it -- push() and pop(). These are templated to allow any type, or rather, any MiniD type or convertible D type. The stack is just a standard stack. You push a value, it's added after the last item on the stack. You pop a value, it removes and returns the last item on the stack. Popping isn't the only way to access items on the stack, however. You can access items directly by using the getParam() method of MDState. This is also templated, and lets you index any value on the stack, starting from 0. The stack grows up, so the first item pushed has an index of 0; the next an index of 1 and so on. Negative indices to mean from the end of the stack are not allowed, because chances are, you're not going to need them nearly as much as in Lua or Squirrel.

However, most of the time when you write native functions there will already be items on the stack. These are the parameters which were passed to the native function. A native function is called with two (native) parameters: the state which is calling this function, and the number of parameters that were passed. There will be exactly as many values on the stack as the number of parameters. You don't have to pop the parameters off the stack, and it's not really efficient to do so. Just leave them on there, and get the parameters with the getParam() method of MDState.

One other thing to point out. MiniD will handle the stack's growth for you automatically. There is no need to check that the stack is big enough before pushing items as in Lua. Just push, and it'll resize the stack as needed.

The other main thing you do with the stack is returning values. Native functions return a (native) integer indicating how many return values it left on the stack. So to return something from a native function, just push the return values and return how many you pushed.

We now know enough to write a simple native function which accepts some parameters, does something with them, and then returns some number of values. Let's write a function which sums all of its arguments, no matter how many arguments it's passed.

int sum(MDState s, uint numParams)
{
	double sum = 0.0;

	for(int i = 0; i < numParams; i++)
		sum += s.getParam!(double)(i);
		
	s.push(sum);
	return 1;
}

This is very straightforward. We just loop through all the parameters we got, get them as doubles, push the sum, and return 1 to mean that we're returning one value. Notice that getParam is templated with double. This will work if the parameter is either a MiniD int or float. The result, being itself a double, will always end up being a MiniD float.

Now let's set up a MiniD context in our native program to register this function and call it.

import minid.minid;
import minid.types;

void main()
{
	MDContext ctx = NewContext();
	ctx.globals["sum"d] = ctx.newClosure(&sum, "sum");
	loadStatementString(ctx.mainThread, "writefln(sum(4.8, 3.2, 16));");
}

int sum(MDState s, uint numParams)
{
	double sum = 0.0;

	for(int i = 0; i < numParams; i++)
		sum += s.getParam!(double)(i);
		
	s.push(sum);
	return 1;
}

Okay, I've dropped a lot on you with this. Let's walk through it.

First we import minid.minid and minid.types. minid.minid is a module which defines some useful helper functions for setting up a MiniD context and running code. minid.types defines most of the native API.

Once in main, we create a new MiniD context with the NewContext? function. By default, all the MiniD standard libraries are loaded into this context, so it's ready to run just about any MiniD code you can throw at it. The context also has a default MDState, the main thread, created for you.

The next line creates a new function closure out of the native function. "ctx.newClosure(&sum, "sum")" means "make a closure out of the native function sum, call it "sum", and make this closure's environment the global namespace." Function environments for native closures usually don't mean much, and especially in this case we don't really care what it is. Besides, we're assigning it to the global "sum" (the left-hand side of the assignment), so having the function's environment be the global namespace makes perfect sense.

The last line in main runs some MiniD code. loadStatementString takes a state to use to execute the compiled code, and a string containing the MiniD source. We're using the main thread of the context as the state. This code, when run, will output "24.0", which is indeed the sum of 4.8, 3.2, and 16. Notice that, as I said before, getting a double parameter works for integer parameters (like 16) as well. Many places in the API will automatically convert integers to floating-point values.

See if you can get this code to compile and run. Once you've done that, go ahead and modify it. Make a min or max function. Make a function which returns multiple arguments, such as one that takes any number of parameters and then returns their squares.

More interesting parameter stuff

MiniD doesn't allow functions to be overloaded, because one name maps to exactly one function closure. What you have to do, then, is write multiple control paths within one function which differ based on the number and types of parameters passed to the function.

Here's an example of a (kind of stupid) function which prints different things based on the type of its first parameter.

import minid.minid;
import minid.types;
import tango.io.Stdout;

void main()
{
	MDContext ctx = NewContext();
	ctx.globals["print"d] = ctx.newClosure(&print, "print");
	loadStatementString(ctx.mainThread, "print(5); print(`hi`); print(6.8);");
}

int print(MDState s, uint numParams)
{
	if(s.isParam!("int")(0))
		Stdout.formatln("Got an int, value is {}", s.getParam!(int)(0));
	else if(s.isParam!("string")(0))
		Stdout.formatln("Got a string, value is \"{}\"", s.getParam!(dchar[])(0));
	else
		Stdout.formatln("Got some other kind of parameter, type is {}", s.getParam(0u).typeString());

	return 0;
}

When run, this outputs:

Got an int, value is 5
Got a string, value is "hi"
Got some other kind of parameter, type is float

Notice the use of the "isParam" templated function. This function is a bit odd. Rather than taking a D type, it takes a MiniD type name as a string. It just returns true or false.

Something else to notice. See in the else clause, I used getParam without a template parameter. getParam defaults to getting an MDValue parameter. I then call typeString on the resulting MDValue to get the name of its type. (I also have to use the "0u" literal as the parameter to getParam because of D's wonky IFTI rules.. ugh).

Something you might be wondering about is -- what if I tried to get a parameter that didn't exist? I'm just throwing around those getParams as if everything is just fine. Shouldn't I do something like "assert(numParams > 0)" at the beginning of the function or something? The answer is: no, getParam will check that you're getting (1) the right type of parameter and (2) a valid parameter index. If I called print without parameters, I'd get a lucid error message, along the lines of "print(native): Bad argument 1: not enough parameters".

Default parameters are easy too. Just create a local variable, and if there are enough parameters, assign it the value of the corresponding parameter; otherwise just assign it your default value.

Calling functions (and classes, and objects, and threads, and...)

Not only do you use the stack for getting parameters to and returning values from your own functions, but you also use it for passing parameters to and getting return values from other functions. But you don't only call functions. You can call classes to instantiate them, you can call objects if they overload the opCall metamethod, and you can call coroutine threads to resume them. All of these are handled the same way, and parallel the way you do them in MiniD.

There are two main ways to call a function: the hard way and the easy way. The hard way takes more code but is more flexible, and the easy way is easy, but because it uses a variadically-templated function, can cause some code bloat if overused. I'll show you the hard way first.

import minid.minid;
import minid.types;
import tango.io.Stdout;

void main()
{
	MDContext ctx = NewContext();
	ctx.globals["callSomething"d] = ctx.newClosure(&callSomething, "callSomething");
	loadStatementString(ctx.mainThread, 
	`function f(x, y)
	{
		writefln("f got ", x, " and ", y);
		return 'a', 3;
	}

	callSomething(f, 5, 10);`);
}

int callSomething(MDState s, uint numParams)
{
	auto cl = s.getParam!(MDClosure)(0);
	
	uint funcSlot = s.push(cl);
	s.push(cl.environment);

	for(uint i = 1; i < numParams; i++)
		s.push(s.getParam(i));

	uint numReturns = s.call(funcSlot, numParams, -1);

	for(int i = numReturns; i > 0; i--)
		Stdout.formatln("Return {}: {}", i, s.valueToString(s.pop()));

	return 0;
}

This outputs:

f got 5 and 10
Return 2: 3
Return 1: a

Let's walk through what's going on with callSomething. This function takes at least one parameter: a function closure to call. After that, any additional parameters are passed through to that function when it's called. It then gets any results, popping them off and displaying them.

The first thing it does is push the closure onto the stack, and stores that stack slot into the funcSlot variable. We'll use this later.

Immediately after pushing the closure, we have to push the context pointer (the 'this' pointer). Every function call has to include the context pointer as the first thing after the function slot. Here we push the function's environment as the context pointer. This is pretty much the default behavior for function calls in the language too.

Then we push any additional parameters which were passed to callSomething. We just do this in a for loop.

Finally we perform the actual call. We use the call method of MDState and pass it the slot of the function we're calling, the number of parameters we pushed onto the stack, and the number of return values we want back. We want all the return values that the function is willing to give us, so we pass in -1. When call returns, all the parameters we passed in are gone off the stack, and in their place are any return values. call returns the number of return values we got back. We loop through all the return values (backwards, of course, because we're popping them off the stack) and print them out.

All function calls work this way. Method calls work the same way, except we just pass the object we're calling the method on as the context pointer (although getting hold of the method closure in the first place could be tricky). We can also use this to call all the other callable types.

This is pretty unwieldy, however, and it would be nice to be able to call a function on one line. Fortunately, we have the easyCall method in MDState to help us.

import minid.minid;
import minid.types;
import tango.io.Stdout;

void main()
{
	MDContext ctx = NewContext();
	ctx.globals["calculate"d] = ctx.newClosure(&calculate, "calculate");
	loadStatementString(ctx.mainThread,
	`function add(x, y)
	{
		return x + y;
	}

	function mul(x, y)
	{
		return x * y;
	}

	writefln(calculate(add, 4, 5));
	writefln(calculate(mul, 4, 5));`);
}

int calculate(MDState s, uint numParams)
{
	auto cl = s.getParam!(MDClosure)(0);
	s.easyCall(cl, 1, MDValue(cl.environment), s.getParam(1u), s.getParam(2u));
	return 1;
}

Here we define a "calculate" function which takes an operation function and two operands. It calls the operation with the two operands, expects one result, and returns that result. easyCall makes this very straightforward. The parameters to it are, in order: the object to call, the number of return values we want, the context pointer to call it with, and then a variadic list of arguments. We expect exactly one result, and since it's already on the stack when easyCall returns, we can just return 1 and that value will become the return value of calculate. This program will output "9" and "20" when run.

The last thing I should show you is how to call methods. Method lookup is a bit tricky, so it's usually not something that you want to do on your own. So MDState provides the callMethod function which takes an object and a method name to call.

void main()
{
	MDContext ctx = NewContext();
	ctx.globals["callSomething"d] = ctx.newClosure(&callSomething, "callSomething");
	loadStatementString(ctx.mainThread,
		`class A
		{
			function foo(x, y)
			{
				writefln("A.foo ", x, ", ", y);
			}
		}
		
		local a = A();
		callSomething(a, "foo");
		`
	);
}

int callSomething(MDState s, uint numParams)
{
	auto context = s.getParam(0u);
	auto cl = s.getParam!(MDString)(1);

	s.callMethod(context, cl, 0, 5, 10);
	return 0;
}

callMethod is very straightforward. It takes the object to call the method on (which will also be used as the 'this' pointer of course), the name of the method to call, the number of returns, and then a list of parameters. This code will display "A.foo 5, 10" when run.

Handling Errors

Error handling in C is a joke. You either have to check return codes, or use a setjmp/longjmp protocol, or check a global errno or something equally tedious and ugly. In D, we have exceptions. In MiniD, we also have exceptions. Actually, as far as the implementation is concerned, these are one and the same.

When MiniD code throws an exception, on the native side it really does throw an exception as well (one derived from MDException, to be exact). The interpreter does some catching and rethrowing trickery to unwind the script call stack, but when it hits a native call, it rethrows the exception to allow native code to catch it. When native code throws an exception, the exact same mechanism is used. This means that any exceptions that script code throws can be caught by native code and vice versa. Isn't that nice?

The interpreter will only bother with exceptions derived from MDException, however. If some other exception occurs, such as an exception in a native library function, or an out of memory error, stack overflow, access violation etc. it will skip over the exception handlers in the interpreter. This is good in most cases, but sometimes you don't want this to happen, such as when you're writing a library wrapper for a native library to expose code to MiniD. In this case, you usually want to translate any non-MiniD exceptions into MiniD exceptions. For this, MDState provides the safeCode method.

safeCode takes a lazy expression of any type. It evaluates that expression, and returns its result. However, if the expression throws a MiniD exception, safeCode will rethrow it, and if the expression throws a non-MiniD exception, safeCode will throw a new MiniD exception whose message is the result of the thrown exception's .toUtf8() method. This way, you can wrap calls to native calls which can throw an exception in safeCode, and any exceptions it throws will be catchable by MiniD code.

As an example, here is the implementation of the string.toInt function in the standard library.

int toInt(MDState s, uint numParams)
{
	dchar[] src = s.getContext!(dchar[]);

	int base = 10;

	if(numParams > 0)
		base = s.getParam!(int)(0);

	s.push(s.safeCode(Integer.toInt(src, base)));
	return 1;
}

You'll notice a new function -- getContext -- in there. That gets the context parameter which was passed to this function. This is how you write methods in native code. The string library is set as the type metatable for strings, and so when you call toInt on a string object, the string on which it was called is passed as the context, which we then retrieve in a similar way to any other parameter.

The optional base parameter is retrieved next, and then it calls the Integer (really tango.text.convert.Integer) toInt function with the string and the base. Note the use of safeCode wrapped around that expression. Integer.toInt can throw exceptions if there is a formatting error or so, and so we use safeCode to ensure that any exceptions that it throws are converted to MiniD exceptions.