Unreal Engine Verse Programming Concepts
Functional Programming, Concurrency and Other Usual Suspects
I gave a brief introduction to Verse programming language in my previous newsletter. As I mentioned there, developers should understand that Verse is heavily influenced by functional programming paradigm, and it also supports other paradigms like
- Determinism
- Expression-oriented structure
- Failure-based control flow
- Speculative execution
- Concurrency
It is obvious that developer should understand how these concepts work. Some of them may sound cryptic, but eventually they have very clear rules. Some of them may be familiar concepts, since they are used in other programming languages, too. Whatever the case, it may require learning new ways to think about program flow, if you want to write efficient, concise, and maintainable Verse code yourself, and understand Verse programs coded by others.
Determinism
Determinism is one of the design goals and key features of the Verse programming language. Determinism means that given the same code and data, the results will always be exactly the same. This can help avoid unexpected or inconsistent behaviour in your programs, especially when dealing with concurrency or multiplayer scenarios.
One way that Verse achieves determinism is by making data immutable by default, which means it cannot be changed after it is created. This prevents data from being modified by different parts of the program or by different threads without your knowledge. If you want to change data, you need to use a variable and explicitly assign a new value to it.
This becomes clearer if you think data and variables in two different ways. Immutable data variables are more like constants, its value has been defined before the execution, and it does not change. Mutable data variables are just a storage space for values which were calculated according to input and functions during runtime. Since the operation of the functions are deterministic, and data cannot change, the current value of a mutable variable is derived from input, immutable variables, and functions only. If the input and other conditions are exactly same, then the value of a mutable variable should also be same.
Another way that Verse achieves determinism is by using failable expressions instead of true/false values to change the flow of your program. Failable expressions are expressions that produce a value if they succeed or don't if they fail. Failable expressions can only be executed in failure contexts, such as if expressions or while loops.
Data and Variables
Mutable and immutable data and variables in Verse are related to the concept of mutability, which is the ability to change the value or data of an object after it is created. Mutable data and variables can be changed, while immutable data and variables cannot.
Some of the differences between mutable and immutable data and variables in Verse are:
Syntax: Data is always declared, defining its type, and then initialized with a value. Mutable data and variables are defined with the keyword `var`, which means you can change their values at any point. Immutable data and variables are defined without the keyword `var`, which means their values are fixed once they are initialized. For example:
var X : int = 10 # mutable variable
X : int = 10 # immutable variable
var X : int # declare a variable X of type int
var Y : int = 10 # declare and initialize a variable Y with the value 10
Semantics: Mutable data and variables allow you to modify their internal state or contained data after creation. Immutable data and variables do not allow this kind of operation. You can only create new objects of the same type with different values. For example:
set X = 20 # valid operation for mutable variable
set X : int = 20 # invalid operation for immutable variable
Scope: Mutable data and variables can only be declared in a function body, where they have local scope. Immutable data and variables can be declared in a function body or in a module, where they have global scope. For example:
myModule := module {
X : int = 10 # immutable variable with global scope
}
myFunction(X:int):int : {
var Y : int = 20 # mutable variable with local scope
…
}
Effects: Mutable data and variables can have side effects, meaning that they can change the state of the program or the environment when they are mutated. Immutable data and variables do not have side effects, meaning that they do not change anything outside their own value. For example:
var Z : int = X + Y # no side effect
set Z = Z + 10 # side effect: changes the value of Z
You can use variables in expressions, such as arithmetic expressions, comparison expressions, or function calls, to perform computations or actions with their values.
Failable Expressions
A failable expression is an expression that can either succeed and produce a value, or fail and produce no value. Failable expressions can only be executed in a failure context, such as an if expression, which defines what happens if the expression fails. For example:
if (Element := MyArray [Index]): # failable expression
Log (Element) # executed if the expression succeeds
else:
Log ("Invalid index") # executed if the expression fails
In this example, the expression `Element := MyArray [Index]` is failable because it tries to assign the value of the array element at the given index to the variable `Element`. If the index is valid, the expression succeeds and produces a value. If the index is invalid, the expression fails and produces no value. The if expression is a failure context that allows the failable expression to be executed, and decides what to do based on whether it succeeds or fails.
The syntax for a failable expression in Verse depends on the type of expression. Some common types of failable expressions are:
Comparison operators: These operators compare two values and produce a value if they succeed or fail if they don't. For example, `x == y` succeeds if `x` and `y` are equal, and fails otherwise. Other comparison operators include `!=`, `<`, `>`, `<=`, `>=`, etc.
Indexing operators: These operators access an element of an array or a map using an index or a key. For example, `MyArray [Index]` succeeds if `Index` is a valid index for `MyArray`, and fails otherwise. Similarly, `MyMap [Key]` succeeds if `Key` is a valid key for `MyMap`, and fails otherwise.
Query: A query expression uses the operator ? and checks whether a logic or option value is true. Otherwise, the expression fails. For example:
if (IsLightOn?):
Light.TurnOff()
else
Light.TurnOn()
Integer Division: For integers, the division operator / is failable, and the result is a rational type if it succeeds.
Function calls: These are expressions that invoke a function with some arguments and produce the return value of the function. For example, `Log (Message)` calls the function `Log` with the argument `Message` and produces no value. A function call can be failable if the function has the effect specifier `<decides>` (and `<transacts>`) in its definition. For example:
IsPuzzleSolved()<decides><transacts> : void =
CurrentState = PuzzleSolution # succeeds if CurrentState is equal to PuzzleSolution, and fails if not
Failure context
As said above, to handle failable expressions in Verse, you need to use a failure context, such as an if expression, a match expression, or a try expression. A failure context defines what happens if the failable expression fails, and allows you to execute other expressions based on the outcome. For example:
if (Element := MyArray [Index]): # failable expression
Log (Element) # executed if the expression succeeds
else:
Log ("Invalid index") # executed if the expression fails
In this example, the if expression is a failure context that handles the failable expression `Element := MyArray [Index]`. If the expression succeeds, the block after the colon is executed. If the expression fails, the block after the else keyword is executed.
Some other examples of failure contexts in Verse are:
not: the operand for not operator in “not expression”
or: the left operand for the or operator in “expression1 or expression2”
for: The iteration expressions and filter expressions in for expressions.
<decides>: The body of a function that has the decided effect specifier.
option: Initializing a variable that has the option type.
Failable Expression vs Exception Handling
A failable expression in Verse and an exception handling in Java (or C# etc.) are two different ways of handling errors or unexpected situations in a program. A failable expression is an expression that can either succeed and produce a value, or fail and produce no value. An exception is an object that represents an event that occurs during the execution of a program that disrupts the normal flow of instructions.
Some of the differences between a failable expression and an exception handling are:
Syntax: A failable expression is part of the syntax of Verse, while an exception in Java is an object that can be thrown and caught using the keywords `throw`, `try`, `catch`, and `finally`. For example:
Verse:
if (Element := MyArray [Index]): # failable expression
Log (Element) # executed if the expression succeeds
else:
Log ("Invalid index") # executed if the expression fails
Java:
try {
Element = MyArray[Index]; // possible exception
System.out.println(Element); // executed if no exception occurs
} catch (ArrayIndexOutOfBoundsException e) {
System.out.println("Invalid index"); // executed if exception occurs
}
Semantics: A failable expression is a form of control flow, meaning that it changes the flow of the program based on whether it succeeds or fails. An exception in Java is a form of error handling, meaning that it indicates an abnormal condition that requires special treatment.
Scope: A failable expression can only be executed in a failure context, such as an if expression, which defines what happens if the expression fails. An exception in Java can be thrown and caught anywhere in the program and can propagate through the call stack until it reaches a matching catch block or the main method.
Effects: A failable expression can have speculative execution, meaning that it can try out actions without committing them, and roll back the effects if it fails. An exception in Java can have side effects, meaning that it can change the state of the program or the environment before it is caught or handled.
Functional Programming
Functional programming is based on the idea of composing functions to solve problems and treats computation as the evaluation of mathematical functions and avoids changing state and mutable data. Functional programming has a strong mathematical foundation and offers many advantages for developing software, but it may feel alien concept, if you have experience with imperative programming languages only.
Functional programming is a programming paradigm where functions are the main building blocks of the code. Functions are treated as first-class citizens, meaning that they can be assigned to variables, passed as arguments, returned from other functions, and stored in data structures. Functions are also pure, meaning that they do not have any side effects or depend on any external state. They only depend on their input parameters and always return the same output for the same input. This makes them easier to reason about, test, and reuse.
Functional programming differs from other paradigms in several ways. For example, functional programming focuses on the "what" of a problem rather than the "how". It uses a declarative style of coding that expresses the logic of the computation without describing its control flow. Functional programming also avoids mutable data and loops, and instead uses immutable data and recursion. This helps to prevent errors caused by accidental modification of shared state or unintended iteration. Functional programming also supports parallel programming, as pure functions can be executed concurrently without any synchronization issues.
You may think that this is just the same as using functions in other programming languages, where they may be called not only functions but subroutines, events etc. In a way they are, but it depends on how you use those functions. For example, if you have defined pure functions in Blueprints, then that is functional programming. Pure function in Blueprint does not change anything, it does not have any side effect. However, if your function makes a change in a value of an Actor variable, then it is not functional programming.
Functions
A function is reusable code that performs an action and produces different outputs based on the input you provide. For example, a function that calculates the area of a circle could look like this:
# A function that takes a radius as an input parameter and returns the area of a circle
AreaOfCircle (Radius : Float) : Float =
# Use the built-in constant Pi and the power operator **
Pi * Radius ** 2
To write a function with two parameters in Verse, you need to follow this syntax:
# Identifier (parameter1 : type, parameter2 : type) <specifier> : type = {}
- The identifier is the name of the function, such as `AreaOfCircle`.
- The parameters are the input variables declared in the function signature and used in the body of the function. You can have zero or more parameters, separated by commas. Each parameter has a name and a type, such as `Radius : Float`.
- The specifier is an optional keyword that specifies how to use or implement a function, such as `public` or `override`.
- The type is the output type of the function, such as `Float`. This means that the function will return a value of that type when it is called. If the function does not return any value, you can use `void` as the type.
- The code block is a block of code that defines what the function does when it is called. It starts with an equal sign `=` and ends with a closing curly brace `}`. The code block can be indented or on the same line as the function signature.
Another example of functional programming in Verse is using higher-order functions, which are functions that can take other functions as arguments or return other functions as results. This allows for more abstraction, modularity, and expressiveness in the code. For example:
# define a function that takes another function as an argument and applies it to two numbers
Apply (F(:int, :int): int, X : int, Y : int) : int =
return F (X, Y)
# define some functions that operate on two numbers
Add (X : int, Y : int) : int = X + Y
Subtract (X : int, Y : int) : int = X - Y
Multiply (X : int, Y : int) : int = X * Y
# use Apply to call different functions with different arguments
Print (Apply (Add, 10, 20)) # prints 30
Print (Apply (Subtract, 10, 20)) # prints -10
Print (Apply (Multiply, 10, 20)) # prints 200
In this example, the function `Apply` is a higher-order function that takes another function `F` as an argument and applies it to two numbers `X` and `Y`. The functions `Add`, `Subtract`, and `Multiply` are some functions that operate on two numbers. The function `Apply` can be used to call different functions with different arguments by passing them as parameters.
Recursive Functions
A recursive function is a function that calls itself directly or indirectly, and it is an important example of functional programming. Recursive functions can be useful for solving problems that have a recursive structure, such as finding the factorial of a number, reversing a string, or solving the Towers of Hanoi puzzle. To write a recursive function in Verse, you need to follow the same syntax as a regular function, but also include a base case and a recursive case.
- The base case is the simplest case of the problem that can be solved without recursion. It usually involves checking a condition and returning a value. The base case is necessary to ensure that the recursion will terminate at some point.
- The recursive case is the case where the problem is reduced to a smaller or simpler subproblem that can be solved by calling the function again with different arguments. The recursive case usually involves calling the function with some modification of the parameters and combining the result with some computation.
Here is an example of how to write a recursive function in Verse that computes the n-th Fibonacci number:
# A function that takes a natural number n as an input parameter and returns the n-th Fibonacci number
Fibonacci (n : int) : int =
# Base case: if n is 0 or 1, return n
if n == 0 or n == 1 then
n
# Recursive case: if n is greater than 1, return the sum of the previous two Fibonacci numbers
else
Fibonacci (n - 1) + Fibonacci (n - 2)
In the beginning, there should be an expression that filters out negative numbers, but I have left it out for the sake of brevity. Recursive functions can be inefficient in terms of both time and space, because they may make repeated or unnecessary function calls and use up memory on the call stack. There are several techniques to improve their efficiency in Verse, such as:
Memoization: This technique involves storing the results of previous function calls in a lookup table (the memo) and returning those results when the function is called again with the same inputs. This can save time by avoiding redundant computations, but it also uses extra space to store the memo. Memoization can be implemented by using a hash table or an array as the memo, and checking for the input value in the memo before making a recursive call. For example, here is a memoized version of the Fibonacci function in Verse:
# A function that takes a natural number n as an input parameter and returns the n-th Fibonacci number
# It uses an array as the memo to store previous results
Fibonacci (n : int) : int =
# Initialize an array of size n + 1 with all elements set to -1
var memo : []int =
for (index := 0..n):
-1
# Define a helper function that performs the recursion and updates the memo
FibonacciHelper (k : int) : int =
# Base case: if k is 0 or 1, return k
If k == 0 or k == 1 then
k
# Recursive case: if k is greater than 1, check the memo first
else
# If the memo already has the result for k, return it
if memo [k] != -1 then
memo [k]
# Otherwise, compute the result recursively and store it in the memo
else
var result : int = FibonacciHelper (k - 1) + FibonacciHelper (k - 2)
memo [k] := result
result
# Call the helper function with n as the argument
FibonacciHelper (n)
Tail recursion: This technique involves writing a recursive function in such a way that the last thing it does is to call itself, without any further computation. This means that there is no need to keep track of the previous function calls on the stack, and the compiler can optimize the recursion into a loop. This can save space by using a constant amount of stack memory, but it may not improve the time complexity. Tail recursion can be implemented by using an accumulator parameter that keeps track of the intermediate result and passes it to the next recursive call. For example, here is a tail-recursive version of the factorial function in Verse:
# A function that takes a natural number n as an input parameter and returns n!
# It uses an accumulator parameter acc to store the intermediate result
# I have left out a check which filters out negative numbers for brevity
Factorial (n : int) : int =
# Define a helper function that performs the tail recursion and updates the accumulator
FactorialHelper (k : int, acc : int) : int =
# Base case: if k is 0, return acc
if k == 0 then
acc
# Recursive case: if k is greater than 0, multiply acc by k and call the helper function with k - 1 and acc as arguments
else
FactorialHelper (k - 1, acc * k)
# Call the helper function with n and 1 as arguments
FactorialHelper (n, 1)
Bottom-up approach: Sometimes the best way to improve the efficiency of a recursive algorithm is to not use recursion at all. In some cases, such as generating Fibonacci numbers, an iterative technique called the bottom-up approach can save both time and space. This technique involves starting from the base case and building up to the desired result using a loop. This can avoid unnecessary function calls and use only a constant amount of memory. For example, here is a bottom-up version of the Fibonacci function in Verse:
# A function that takes a natural number n as an input parameter and returns the n-th Fibonacci number
# It uses a bottom-up approach to compute the result iteratively
Fibonacci (n : int) : int =
# Base case: if n is 0 or 1, return n
if n == 0 or n == 1 then
n
# Iterative case: if n is greater than 1, use a loop to calculate the result from bottom up
else
# Initialize two variables to store the previous two Fibonacci numbers
var prevPrev : int = 0 # F(0)
var prev : int = 1 # F(1)
# Initialize a variable to store the current Fibonacci number
var current : int = 0
# Loop from 2 to n, updating the variables at each iteration
for (i := 2 .. n):
# Calculate the current Fibonacci number as the sum of the previous two
current := prevPrev + prev
# Update the previous two Fibonacci numbers
prevPrev := prev
prev := current
# Return the current Fibonacci number as the result
current
Speculative Execution
Speculative execution in Verse is the ability to try out actions without committing them. This means that you can execute a series of expressions that may change the state of your program, such as assigning values to variables or calling functions, but those changes will be undone if any of the expressions fail. This way, you can ensure that your program is always in a consistent and valid state, even if you encounter failures.
If you have ever programmed SQL databases, and used transactions and rollback, then you already know how this works in principle. In SQL databases you can make complex transactions, inserting new data and altering old data as one action, but if one of those operations fail, then all operations are rollbacked, and database values are the same as they were before you tried that transaction. Otherwise, all operation results are committed to database.
Speculative execution can only happen within a failure context, such as an if expression. A failure context allows you to execute failable expressions, which are expressions that can either succeed and produce a value, or fail and produce no value. For example:
if (Element := MyArray [Index]): # failable expression
Log (Element) # executed if the expression succeeds
else:
Log ("Invalid index") # executed if the expression fails
In this example, the if expression is a failure context that executes the failable expression `Element := MyArray [Index]`. If the expression succeeds, the variable `Element` is assigned the value of the array element at the given index, and the block after the colon is executed. If the expression fails, the assignment is rolled back, and the block after the else keyword is executed.
Concurrency
Concurrency and parallelism are two related but distinct concepts in computer science. Concurrency is the task of running and managing multiple computations at the same time, while parallelism is the task of running multiple computations simultaneously.
Concurrency does not necessarily imply parallelism, as concurrent tasks can be executed in an interleaved manner on a single processor by switching between them. For example, multitasking on a single-core machine is a form of concurrency, but not parallelism. Concurrency is achieved by using techniques such as threading, multiprocessing, distributed computing, or coroutines.
Parallelism does imply concurrency, as parallel tasks are also concurrent tasks that can start, run, and complete in overlapping time periods. However, parallelism requires multiple processors or cores to execute the tasks at the same time. For example, matrix multiplication on a multicore processor is a form of parallelism and concurrency. Parallelism is achieved by using techniques such as multiprocessing, distributed computing, vectorization, or pipelining.
The main goal of concurrency is to increase the efficiency and responsiveness of a system by minimizing the idle time of the processor. The main goal of parallelism is to increase the performance and speed of a system by utilizing the available resources.
Concurrency in Verse is the ability to perform actions simultaneously using language-level concurrency instead of relying on system-level threads across multiple processors. This means that you can write code that can execute in parallel without worrying about the low-level details of how the processor handles it.
Concurrency in Verse is achieved by using concurrency expressions, which are expressions that determine whether other expressions execute concurrently (at the same time), or in sequence (one after another). Concurrency expressions are especially useful when you’re using async expressions, which are expressions that can take time to evaluate and may not complete in the current simulation update.
A function with the effect specifier `<suspends>` is a function that can suspend and cooperatively transfer control to other concurrent expressions at various points over several simulation updates before it completes. This means that the function can execute in an async context, which is the body of another function that has the `<suspends>` effect specifier. For example:
OnBegin<override> ()<suspends> : void = # async context
HideAllPlatforms () # function call with <suspends> effect
HideAllPlatforms ()<suspends> : void = # function definition with <suspends> effect
for (Platform : Platforms):
Platform.Hide ()
Sleep (Delay) # async expression that suspends the function
In this example, the function `HideAllPlatforms` has the `<suspends>` effect specifier in its definition, which means that it can suspend and resume its execution over time. The function is called from another function, `OnBegin`, that also has the `<suspends>` effect specifier in its signature, which means that it can execute async expressions, such as function calls or sleep expressions. The function `HideAllPlatforms` uses a for loop to hide each platform in an array, and then suspends its execution for a given delay using the sleep expression. After the delay, the function resumes its execution and hides the next platform, until all platforms are hidden.
This example function moves a character to the nearest NPC in the world, and suspends its execution until the movement is completed:
MoveToNearestNPC ()<suspends> : NPC =
var NearestNPC := FindNearestNPC ()
MoveTo (NearestNPC) # async expression that suspends the function
return NearestNPC
Another example of concurrency in Verse is using the sync expression, which runs two or more async expressions concurrently and waits for all of them to complete before producing a value. For example:
sync: # concurrency expression
Sleep (1.0) # async expression
Log ("Hello") # async expression
In this example, the sync expression runs the sleep expression and the log expression at the same time, and waits for both of them to finish before producing no value. The sleep expression suspends the execution for 1 second, while the log expression prints "Hello" to the console.
race expression: This expression runs two or more async expressions concurrently and only waits for one expression to complete before producing a value. The other expressions are cancelled and their effects are rolled back. For example:
race: # concurrency expression
Sleep (1.0) # async expression
Log ("Hello") # async expression
In this example, the race expression runs the sleep expression and the log expression at the same time, but only waits for one of them to finish before producing no value. The log expression finishes first and prints "Hello" to the console, while the sleep expression is cancelled and its effect is undone.
branch expression: This expression starts one or more async expressions, then immediately executes the following expressions without waiting for them to complete. For example:
branch:
# This block continues until completed
AsyncFunction1() # Starts effectively the same time as AsyncFunction3()
Method1() # Block can be mixed with immediate expressions
AsyncFunction2()
AsyncFunction3() # Starts effectively the same time as AsyncFunction1()
# If branch block task is still running when AsyncFunction3 completes
# then any remaining branch task is canceled
This is similar to unstructured concurrency. When using branch you must understand that the branch blocks may not be executed completely.
rush expression: This expression runs two or more async expressions concurrently. When the fastest subexpression completes, any expression that follows the rush is evaluated, and any remaining subexpressions continue to evaluate. For example:
set WinnerResult = rush:
# All three async functions start at the same time
AsyncFunctionLongTime()
AsyncFunctionShortTime() # This will win and its result is used
AsyncFunctionMediumTime()
# Next expression is called after the fastest async function (AsyncFunctionShortTime()) completes.
# All other subexpression tasks (AsyncFunctionLongTime(), AsyncFunctionMediumTime()) continue.
NextExpression(WinnerResult)
spawn expression: The spawn expression starts one async function invocation, and any expression that follows the spawn is executed immediately while the started async function task continues independently until it completes. Spawn expression is allowed outside of an async context. For example:
spawn{MyAsyncFunction()} # started at same time as otherExpression
otherExpression # started at same time as MyAsyncFunction
Task: A task is an object used to represent the state of a currently-executing async function. Task objects are used to identify where an async function is suspended, and the values of local variables at that suspend point.
Tasks execute concurrently in a cooperatively multitasked environment. A task can be durational, based on a lifespan of one or more updates before it completes. Tasks can be sequential, overlapped, staggered, and so on, in any logical order. The sequence and overlapping flow of tasks is specified through the use of structured or unstructured concurrency expressions.
Each task can be concurrently arranged sequentially, overlapped, staggered, and so on, in any logical order of time. Internally, a task could have a caller (or even several callers), and zero or more dependent sub-tasks that form a call graph (as opposed to a call stack).
You may notice similarity with threads, but there are differences. Context switching between tasks does not involve any system calls etc. You do not need mutexes or semaphores, either.
The task(t:type) class allows direct programmatic querying and manipulation of tasks in an unstructured manner, though it is generally recommended that tasks be manipulated through structured concurrency expressions for greater clarity, power and efficiency.
Currently, the only exposed function for task is Await(), which waits until the current task has completed. This essentially anchors a task and adds a caller for it to return to at the call point.
spawn{AsyncFunction3()}
# Get task to query / give commands to
# starts and continues independently
Task2 := spawn{Player.MoveTo(Target1)}
Sleep(1.5) # Wait 1.5 Seconds
MyLog.Print("1.5 Seconds into Move_to()")
Task2.Await() # wait until MoveTo() completed
Wait(0.5) # Wait 0.5 Seconds
# Explicit start and wait until completed
# Task1 could still be running
Target1.MoveTo(Target2)
Summary
As you can see, Verse programming is quite different than basic C++ and Blueprint programming in Unreal Engine, and we have only scratched the surface. If you want to utilize Verse programming to its fullest, according to its design principles, you must learn to think and design your UE applications in a different way. Whether you learn it or not, depends entirely on your attitude. For example, concurrency needs lots of practice to master it – understanding what is executing, when it is executing and when not. Combined with speculative execution it can look for an untrained eye that something did execute, but in reality, it was rolled back.
During my professional and academic career, I have noticed that there are too many developers who stubbornly think that some old paradigm is better than others and stick to it adamantly, refusing to try anything else. Sometimes it may be better, but sometimes it is just prejudice without even trying out what that new way could offer.
It is also question of subjective and objective views. In my experience there are very few universally and objectively best solutions, because usually they are matters of opinion, and we value them according to our own workflows and preferences. What works for me, may not work for you, and vice versa. But in any case, I always urge people to even try new things. There are no obligations, no strings attached… if you don’t like it after carefully examining it, then just throw it away. But even then, you may have learned something new.