RAAI 2019 - A Hands-on Laboratory Tutorial on Using CST to build a Cognitive Architecture

Main Goal

The main goal of this hands-on tutorial is to have a practical experience on using CST to build a cognitive architecture to control a virtual creature at the WorldServer3D virtual environment, in order to locate food and jewels, and generate the creature behavior such that these tasks can be acomplished by means of the cognitive architecture. In order to fulfill this goal, we first introduce a demo code, where the basics of CST are illustrated, and then propose an exercise where this code is supposed to be enhanced such that new goals can be achieved.


In order to run the tutorial, you need the following software installed in your computer:

  • Java JDK 8 ... if you already have the JDK 8 installed in your computer, you don't need this. If you are not allowed to use new Oracle's license in your computer, you might install OpenJDK8 instead. The WS3D application does not run on JDK 9 and greater.
  • NetBeans 11 you might need to install the Gradle and Groovy plugins to compile the CST source.
  • WS3D Source Code - The WorldServer3D Virtual Environment: (github), (zip)
  • WS3DProxy Source Code - A Java Proxy for sensing and controlling a WS3D simulation: (github), (zip)
  • DemoCST Source Code - A CST Demo Application Controlling a creature in WS3D: (github), (zip)
  • CST Source Code - The Cognitive Systems Toolkit: (github), (zip)

Activity 1 - Understand the Demo code

The purpose of this activity is to browse the provided demo code and understand its behavior. The code illustrates the development of an artificial mind for an artificial creature living in the WorldServer3D virtual environment, and in its original form just locate and pick apples from the environment.

Understanding the WorldServer3D Environment

If you still didn't installed the WS3D, do it now. Download the source code of WS3D and open it in a known folder in your PC. Then open it with Netbeans.

When you run the WS3D application, a console like the one in figure below should appear.

You might then start including creatures, apples, nuts, jewels and blocks in the environment. Explore and discover how to do it. After some inclusions, you might have a scenario like this:

The WS3D environment works by means of a socket connection in port 4011. This means that to control a creature in WS3D, you must open a TCP/IP connection in localhost port 4011 and then start sending commands to the environment. If you have access to a shell, you might try a "telnet 4011" and start sending commands directly to WS3D. The commands WS3D accepts can be seen at the code of worldserver3d.Main.java class.  You can send these commands from a program written in any language supporting socket connections.

Understanding the WS3DProxy

Alternatively, we have developed a Java proxy for creating creatures, apples, nuts, jewels and blocks, which makes this task easier if you are making your program in Java, so you don't need to bother with socket issues and just start controlling WS3D directly from Java. The WS3DProxy just wraps everything within a socket infrastructure and send the commands through sockets for you.

Let's start knowing a little bit more about the WS3DProxy. If you still didn't downloaded the source code from GitHub (see the link at Preparations), do it now.

The most important classes at the WS3DProxy are the WS3DProxy class, used to start the proxy, the  World class, which gives a reference to the WS3D environment, where you can create things at the environment, and the Creature class, from which you can read the creature sensors and send commands to its actuators. You might now take some time to browse the source code of WS3DProxy in order to evaluate what you have at your disposal.

At this point, we recommend you try to create a small program to initialize an environment, create some items on it and control your creature with the keyboard.

Understanding the WS3DApp code

The WS3DApp provides you with a complete demo, using CST to control a creature at WS3D. Basically, we will be contruction the following simple cognitive architecture:

In order to understand how it works follow the CST Core Model Trail. Besides that, download the code of the WS3DApp and study it in order to understand how it works.

Activity 2 - Extend the Demo code to look for both Crystals and Food

Based on the demo code from the Core Model Trail, develop now a mechanism for searching for crystals and food, meeting the following specifications: while the energy resources are higher than 40%, the creature should only collect food if apples and nuts are obstructing their way. In this situation, the creature should search for the crystals necessary to complete their leaflet and trade tese crystals for points. If the energy level decreases below 40%, the creature should look for the closest food they are aware and just collecting crystals if they are obstructing their way.

You might notice that in our WS3DApp demo, a very peculiar situation holds: the hands motor codelet receives commands from just one behavior codelet: the EatClosestApple - the legs motor codelet, though, receives commands from two distinct behavior codelets: the GoToClosestApple codelet and the Forage codelet. This could cause a potential problem, if both codelets suggest different commands at each time: the legsMO might just hold the last command sent by the behavior codelets, and this might cause a running condition, a known problem in multi-thread programming. Fortunately, in our example, this doesn't happen, because the Forage codelet only suggests an action if the list of known apples is empty, and the the GoToClosestApple only suggests an action if there is at least one known apple. This creates a mutual exclusion situation wich avoids the running condition to happen. But now, while extending our source code to other behaviors, we cannot count with this fortunate situation anymore, we need to solve this. In order to solve this, CST provides the MemoryContainer class, which is basically a boosted MemoryObject, where multiple codelets might write at the same time, and the MemoryContainer will hold separate objects for input, and a standard mechanism for output. The mechanism of a MemoryContainer can be understood by the figure below: