AI Vision Sensor

Introduction

The AI Vision Sensor enable robots to detect and track visual information in their environment. By identifying objects, colors, and patterns, the AI Vision Sensor allows robots to analyze their surroundings and respond to what they see.

For the examples below, the configured AI Vision Sensor will be named AIVision1, and the configured Color Signature objects, such as redBox, will be used in all subsequent examples throughout this API documentation when referring to aivision class methods.

Below is a list of all methods:

Getters

• takeSnapshot – Captures data for a specific Color Signature or Color Code.

• objectCount – Returns the number of detected objects as an integer.

• largestObject – Immediately select the largest object from the snapshot.

Properties – Object data returned from takeSnapshot.

• .exists – Whether the object exists in the current detection as a Boolean.

• .width – Width of the detected object in pixels.

• .height – Height of the detected object in pixels.

• .centerX – X position of the object’s center in pixels.

• .centerY – Y position of the object’s center in pixels.

• .angle – Orientation of the Color Code in degrees.

• .originX – X position of the object’s top-left corner in pixels.

• .originY – Y position of the object’s top-left corner in pixels.

• .id – Classification or tag ID of the object.

• .score – Confidence score for AI Classifications (1–100).

Constructors – Manually initialize and configure the sensors.

• aivision – Creates an AI Vision Sensor.

• aivision::colordesc – Creates a Color Signature.

• aivision::codedesc – Creates a Color Code.

takeSnapshot

takeSnapshot captures an image from the AI Vision Sensor, processes it based on the signature, and updates the objects array. This method can also limit the amount of objects captured in the snapshot.

Color Signatures and Color Codes must be configured first in the Vision Utility before they can be used with this method.

The objects array stores objects ordered from largest to smallest by width, starting at index 0. Each object’s properties can be accessed using its index. objects is an empty array if no matching objects are detected.

Default Usage:
AIVision1.takeSnapshot(signature)

Overloads:
AIVision1.takeSnapshot(signature, count)

Parameters	Description
signature	What signature to get data of. Color Signatures or Color Codes – The name of the AI Vision Sensor, two underscores, and then the Color Signature’s or Color Code’s name.For example: AIVision1__redBox. aivision::ALL_AIOBJS – Cubes, rings, and balls. aivision::ALL_TAGS – All AprilTags.
count	Optional. Sets the maximum number of objects that can be returned from 1 to 24 (default: 8).

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Move forward if an object is detected
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);
    if (AIVision1.objects[0].exists) {
      Drivetrain.driveFor(forward, 50, mm);
    }
    wait(50, msec);
  }
}

Color Signatures

A Color Signature is a unique color that the AI Vision Sensor can recognize. These signatures allow the AI Vision Sensor to detect and track objects based on their color. Once a Color Signature is configured, the sensor can identify objects with that specific color in its field of view. Color signatures are used with takeSnapshot to process and detect colored objects in real-time.

In order to use a configured Color Signature in a project, its name must be the name of the sensor, two underscores, and then the Color Signature’s name. For example: AIVision1__redBox.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();

  while (true) {
    Brain.Screen.setCursor(1, 1);
    Brain.Screen.clearLine(1);
    // Change to any configured Color Signature
    AIVision1.takeSnapshot(AIVision1__redBox);
    if (AIVision1.objects[0].exists) {
      Brain.Screen.print("Color detected!");
    }
    wait(50, msec);
  }
}

Color Codes

A Color Code is a structured pattern made up of color signatures arranged in a specific order. These codes allow the Vision Sensor to recognize predefined patterns of colors. Color codes are useful for identifying complex objects or creating unique markers for autonomous navigation.

To use a configured Color Code in a project, its name must be passed as a string in the format: the Vision Sensor’s name, followed by two underscores, and then the Color Code’s name. For example: AIVision1__redBlue.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();

  while (true) {
    Brain.Screen.setCursor(1, 1);
    Brain.Screen.clearLine(1);
    // Change to any configured Color Code
    AIVision1.takeSnapshot(AIVision1__redBlue);
    if (AIVision1.objects[0].exists) {
      Brain.Screen.print("Code detected!");
    }
    wait(50, msec);
  }
}

largestObject

largestObject retrieves the largest detected object from the objects array.

This method can be used to always get the largest object from objects without specifying an index.

Default Usage:
AIVision1.largestObject

Parameters	Description
	This method has no parameters.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display the closest AprilTag's ID
  while (true) {
    Brain.Screen.setCursor(1, 1);
    Brain.Screen.clearLine(1);
    AIVision1.takeSnapshot(aivision::ALL_TAGS);
    if (AIVision1.objects[0].exists) {
      Brain.Screen.print("%d", AIVision1.largestObject.id);
    }
    wait(50, msec);
  }
}

objectCount

objectCount returns the number of items inside the objects array as an integer.

Default Usage:
AIVision1.objectCount

Parameters	Description
	This method has no parameters.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display the number of detected objects
  while (true) {
    Brain.Screen.setCursor(1, 1);
    Brain.Screen.clearLine(1);
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);
    if (AIVision1.objects[0].exists) {
      Brain.Screen.print("%d", AIVision1.objectCount);
    }
    wait(50, msec);
  }
}

installed

installed returns an integer indicating whether the AI Vision Sensor is currently connected to the IQ (2nd gen) Brain.

• 1 – The AI Vision Sensor is connected to the IQ (2nd gen) Brain.

• 0 – The AI Vision Sensor is not connected to the IQ (2nd gen) Brain.

Parameters	Description
	This method has no parameters.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display a message if the AI Vision Sensor is detected
  if (AIVision1.installed()){
    Brain.Screen.print("Installed!");
  }
}

objects

objects returns an array of detected object properties. Use the array to access specific property values of individual objects.

Default Usage:
AIVision1.objects

Properties

There are ten properties that are included with each object stored in the objects array after takeSnapshot is used.

Some property values are based off of the detected object’s position in the Vision Sensor’s view at the time that takeSnapshot was used. The AI Vision Sensor has a resolution of 320 by 240 pixels.

.exists

.exists returns an integer indicating if the index exists in the objects array or not.

• 1: The index exists.

• 0: The index does not exist.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Move forward if an object is detected
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);
    if (AIVision1.objects[0].exists) {
      Drivetrain.driveFor(forward, 50, mm);
    }
    wait(50, msec);
  }
}

.width

.width returns the width of the detected object in pixels, which is an integer between 1 and 320.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Approach an object until it's at least 100 pixels wide
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);

    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].width < 100) {
        Drivetrain.drive(forward);
      } else {
        Drivetrain.stop();
      }
    } else {
      Drivetrain.stop();
    }
    wait(50, msec);
  }
}

.height

.height returns the height of the detected object in pixels, which is an integer between 1 and 240.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Approach an object until it's at least 90 pixels tall
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);

    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].height < 90) {
        Drivetrain.drive(forward);
      } else {
        Drivetrain.stop();
      }
    } else {
      Drivetrain.stop();
    }
    wait(50, msec);
  }
}

.centerX

.centerX returns the x-coordinate of the detected object’s center in pixels, which is an integer between 0 and 320.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Turn until an object is directly in front of the sensor
  Drivetrain.setTurnVelocity(10, percent);
  Drivetrain.turn(right);
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);

    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].centerX > 140 && AIVision1.objects[0].centerX < 180) {
        Drivetrain.stop();
      }
    }

    wait(10, msec);
  }
}

.centerY

.centerY returns the y-coordinate of the detected object’s center in pixels, which is an integer between 0 and 240.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Approach an object until it's at least 150 pixels tall
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);

    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].centerY < 150) {
        Drivetrain.drive(forward);
      } else {
        Drivetrain.stop();
      }
    } else {
      Drivetrain.stop();
    }
    wait(50, msec);
  }
}

.angle

.angle returns the orientation of the detected Color Code or AprilTag in degrees, which is a double between 0 and 360.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Turn left or right depending on how a configured
  // Color Code is rotated
  while (true) {
    AIVision1.takeSnapshot(AIVision1__redBlue);
    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].angle > 50 && AIVision1.objects[0].angle < 100) {
        Drivetrain.turn(right);
      }
      else if (AIVision1.objects[0].angle > 270 && AIVision1.objects[0].angle < 330) {
        Drivetrain.turn(left);
      }
      else {
        Drivetrain.stop();
      }
    } else {
      Drivetrain.stop();
    }
    wait(50, msec);
  }
}

.originX

.originX returns the x-coordinate of the top-left corner of the detected object’s bounding box in pixels, which is an integer between 0 and 320.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display if an object is to the left or the right
  while (true) {
    Brain.Screen.clearScreen();
    Brain.Screen.setCursor(1, 1);
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);
    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].originX < 120) {
        Brain.Screen.print("To the left!");
      } else {
        Brain.Screen.print("To the right!");
      }
    } else {
      Brain.Screen.print("No objects");
    }
    wait(100, msec);
  }
}

.originY

.originY returns the y-coordinate of the top-left corner of the detected object’s bounding box in pixels, which is an integer between 0 and 240.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display if an object is close or far
  while (true) {
    Brain.Screen.clearScreen();
    Brain.Screen.setCursor(1, 1);
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);
    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].originY < 110) {
        Brain.Screen.print("Close");
      } else {
        Brain.Screen.print("Far");
      }
    }
    wait(100, msec);
  }
}

.id

.id returns the ID of the detected AI Classification or AprilTag as an integer.

AI Classification	ID
Blue Ball	0
Green Ball	1
Red Ball	2
Blue Ring	3
Green Ring	4
Red Ring	5
Blue Cube	6
Green Cube	7
Red Cube	8

For an AprilTag, the .id property represents the detected AprilTag’s ID number in the range of 0 to 36. For an AI Classification, the id corresponds to the predefined id as shown below.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Move forward when AprilTag 1 is detected
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_TAGS);
    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].id == 1) {
        Drivetrain.drive(forward);
      }
    } else {
      Drivetrain.stop();
    }
    wait(50, msec);
  }
}

.score

.score returns the confidence score of the detected AI Classification as an integer between 1 and 100.

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display if a score is confident
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);

    if (AIVision1.objects[0].exists) {
      if (AIVision1.objects[0].score > 95) {
        Brain.Screen.print("Confident");
      } else {
        Brain.Screen.print("Not confident");
      }
    }

    wait(50, msec);
  }
}

Constructors

Constructors are used to manually create aivision, colordesc, and codedesc objects, which are necessary for configuring the AI Vision Sensor outside of VEXcode. If fewer arguments are provided, default arguments or function overloading should be used in the constructor definition.

AI Vision Sensor

aivision creates an AI Vision Sensor.

Default Usage:
aivision( int32_t index, uint8_t bright, Args &... sigs )

Parameters	Description
port	Which Smart Port the AI Vision Sensor is connected to, from 1 to 12.
sigs	Optional. The name of one or more signatures: Color Signature Color Code aivision::ALL_TAGS – All AprilTags aivision::ALL_AIOBJS – All AI Classifications

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Move forward if an object is detected
  vex::aivision AIVision1(PORT11, aivision::ALL_AIOBJS);
  while (true) {
    AIVision1.takeSnapshot(aivision::ALL_AIOBJS);
    if (AIVision1.objects[0].exists) {
      Drivetrain.driveFor(forward, 50, mm);
    }
    wait(50, msec);
  }
}

Color Signature

colordesc creates a Color Signature. Up to seven different Color Signatures can be stored on an AI Vision Sensor at once.

Default Usage:
colordesc(index, uMin, uMax, uMean, vMin, vMax, vMean, rgb, type)

Parameter	Description
index	The colordesc object’s index, from 1 - 7. Note: Creating two colordesc objects with the same index number will cause the second created object to override the first.
uMin	The value from uMin in the Vision Utility.
uMax	The value from uMax in the Vision Utility.
uMean	The value from uMean in the Vision Utility.
vMin	The value from vMin in the Vision Utility.
vMax	The value from vMax in the Vision Utility.
vMean	The value from vMean in the Vision Utility.
rgb	The value from rgb in the Vision Utility.
type	The value from type in the Vision Utility.

In order to obtain the values to create a Color Signature, go to the Vision Utility. Once a Color Signature is configured, copy the parameter values from the Configuration window.

aivision::colordesc AIVision1__greenBox(1, 81, 149, 44, 17, 0.23);
vex::aivision AIVision1(PORT1, AIVision1__greenBox);

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display if a green object is detected
  while (true) {
    Brain.Screen.setCursor(1, 1);
    Brain.Screen.clearLine(1);
    // Change to any configured Color Signature
    AIVision1.takeSnapshot(AIVision1__greenBox);
    if (AIVision1.objects[0].exists) {
      Brain.Screen.print("Color detected!");
    }
    wait(50, msec);
  }
}

Color Code

codedesc creates a Color Code. It requires at least two already defined Color Signatures in order to be used. Up to eight different Color Codes can be stored on an AI Vision Sensor at once.

Default Usage:
codedesc(sig1, sig2)

Overloads:

• codedesc(sig1, sig2, sig3)

• codedesc(sig1, sig2, sig3, sig4)

• codedesc(sig1, sig2, sig3, sig4, sig5)

Parameters	Description
sig1	A previously created signature object, or the int32_t index of a previously created signature object.
sig2	A previously created signature object, or the int32_t index of a previously created signature object.
sig3	A previously created signature object, or the int32_t index of a previously created signature object.
sig4	A previously created signature object, or the int32_t index of a previously created signature object.
sig5	A previously created signature object, or the int32_t index of a previously created signature object.

aivision::colordesc AIVision1__greenBox(1, 85, 149, 46, 23, 0.23);
aivision::colordesc AIVision1__blueBox(2, 77, 135, 125, 27, 0.29);
aivision::codedesc AIVision1__greenBlue(1, AIVision1__greenBox, AIVision1__blueBox);
vex::aivision AIVision1(PORT1, AIVision1__greenBox, AIVision1__blueBox, AIVision1__greenBlue);

int main() {
  // Initializing Robot Configuration. DO NOT REMOVE!
  vexcodeInit();
  // Display if greenBlue code is detected
  while (true) {
    Brain.Screen.setCursor(1, 1);
    Brain.Screen.clearLine(1);
    // Change to any configured Color Signature
    AIVision1.takeSnapshot(AIVision1__greenBlue);
    if (AIVision1.objects[0].exists) {
      Brain.Screen.print("Code detected!");
    }
    wait(50, msec);
  }
}