How can AI recognise what it sees?

AI real world applications

1. Use a suitable learning hook that will engage students in a task of introducing the idea that a computer can see using an inbuilt camera and a field of AI called computer vision. Ideas may include:
   1. Shark spotting: A drone fitted with a camera and using an AI system can spot a shark and send data to beach patrols to warn swimmers in the area. How might it tell a shark from a dolphin?
   2. View the video to see how an AI is 90% accurate in spotting crocs along a river in Queensland. How might it distinguish a crocodile from a fallen tree branch? Find out more, view this article Croc spotting.
   3. Weed spotting in Kakadu National Park: View the video about a project that combines Indigenous knowledge and AI technology. Para grass, an invasive weed, is choking precious wetlands. With the help of drones and AI technology, Traditional Owners are able to target these weeds and manage country. How does the AI in the drone know a weed from a wetland plant?
   4. Waymo's fully self-driving cars: View this video for a sense of what it’s like to be a passenger in a driverless car. How does the car distinguish a traffic light from a tree?
   5. An example of AI classified images. Use these images to show the range of objects an AI can see and classify. The boxes indicate that the AI has detected objects and classified them.
   6. View the video explaining the Seeing AI app from Microsoft that is designed for the blind and people with low vision. Seeing AI demonstrates how technology can make the world more inclusive. The app uses AI to narrate the world around the user describe nearby people, text and objects. Download the app and explore its use or alternatively view this review of Seeing AI app.

Unplugged: Feature extraction

2. One way an AI recognises an object is by feature extraction. These are the features that help the AI differentiate one object from another. Take one of the examples below; for example, shark, crocodile or weed-spotting or traffic light detection. Ask students to describe features of each to enable the AI to recognise one object from another.

   Shark	Dolphin
   Crocodile	Tree branch
   Weed	Wetland plant
   Traffic light	Tree

Additional scaffolding:

Some students may need individual support during this activity to select and view relevant images and discuss features of each.

An AI using feature extraction

3. Now take that idea of feature extraction a step further. Ask students to think about the features of the objects they described and how they might represent these in a simple line drawing.

   A fun way to do this is to go to the AutoDraw website and select Launch an experiment. Ask students to draw a shark, crocodile, weed or traffic light onscreen first, and see how well the AI guesses (predicts) what they are drawing.

   Ask:

   • Can the AI guess what you are drawing? What features do you think it is using to make its prediction? (Hopefully these are the same ones you identified.)
   • To make its prediction, the AI uses a huge bank of line drawings already labelled by a human.
   • What close, but not quite correct, predictions does the AI make? Why might this happen? (Share similar features that might confuse the AI.)

   You might also try out Quick draw. It will direct students to draw a specific object. The AI will guess what it is, based on known patterns in how people from all over the world draw.

Breaking an object down into shapes

4. Computers powered by AI also ‘look’ at and categorise objects by breaking them down into shapes. We can model this process by creating a simple representation of an object using geometric shapes.

   As a class view the slide deck: AI sees shapes in presentation mode which shows how a computer might ‘see’ an image as a collection of shapes. It shows an animated sequence of shapes arranged to represent an animal. As it is animated students can guess the animal.

   Ask students to try representing an animal or object using common geometrical shapes. They could use shape blocks, cut out shapes from cardboard or use familiar presentation software that has animate function such as Keynote for iOS or PowerPoint for Windows. Have students work with a partner to see if they can identify their partner’s object. What were the key features that helped identify the animal or object?

   

   Image: This is how a computer might 'see' the kangaroo made up of different shapes.

What does a computer actually see?

5. In reality, a computer sees images as individual picture elements (pixels). Use this pixel viewer to show that an image is made up of pixels. Each colour pixel in the image is made up of numbers that represent the colour as a combination of Red-Green-Blue (RGB). So the computer recognises a kangaroo, for example, based on a pattern of pixels and complex mathematical algorithms.

   No wonder it is such a challenge for a computer to recognise an object accurately!

   

   Image: Pixel viewer screen capture: Kangaroo (Right Image Zoom in showing RGB for each pixel)

An AI tool that detects and classifies objects

6. Explore an AI that detects objects and classifies them. Use Google’s Vision AI which uses pre-trained machine learning models to assign labels to images and quickly classify them into millions of predefined categories. Detect objects and faces, read printed and handwritten text. Note the confidence level the AI has of its match and predicted classification. In the example the AI is 80% confident that the object detected is a kangaroo.

   Have students begin by dragging a photo into the API.

   • Does the API correctly classify the object in the image?
   • How sure or confident is the AI in its prediction?

    

   Students can test a new image by selecting ‘new file’ and uploading an image from their computer. You may want to download these images, which are copyright free.

   Discuss the level of accuracy of the AI and its confidence level. To help understand percentage you can draw a bar and indicate the level coloured in. 50% half coloured in, 80% as 4/5 coloured in etc. The closer to 100% the higher the confidence the AI has in its predicted match. Discuss the training data and that the AI may only have been trained on common Australian animals such as Kangaroo, Koala and that it would predict an emu as an ostrich as this is what it most likely was trained on.

   

   Kangaroo image credit: pen_ash/pixabay

Classification by features

7. Introduce the idea of classification using the sorting of Animal picture cards. The cards can be sorted into groupings such as:

   1. common characteristics such as wings/no wings, legs/no legs, body covering (hair/fur, moist skin, scales or feathers)
   2. animal type: mammal, reptile, bird, amphibian or fish
   3. how it moves.

   Share the different ways the animals were sorted.

   Ask students to look at the features of animals and work out how these can be used to classify the animals. Create a table of data to describe the key features of four to five Australian animals. Alternatively, use a combination of images with text/numbers.

Image: Information in a Table and presented visually

A spreadsheet is a useful way to organise the information, as it can be filtered by column heading. You could use this Excel spreadsheet or this Google spreadsheet for students not familiar with setting up a spreadsheet. It has four tabs.

• An table set up for students to enter data for example yes/no.
• Select yes/no: Use this tab for students to select yes/no against the listed features.
• Conditional formatting: Shows yes/no formatted for students to sort and filter.
• Stacked column chart: Shows data as numerical data (1 or 0) instead of yes or no. The data can be filtered by animal to view the features selected.

Exploring an AI that classifies an animal by features

8. Use this experiment to show how an AI identifies a particular animal based on selected features. The AI has been trained to identify to identify six relevant animals by whether each animal has features such as legs, arms, wings or fins, tail or no tail, certain types of body covering and external ears or ear holes.

   Note the level of black in the text box in the output layer. This indicates how sure the AI is in its prediction. Black indicates a high confidence level. Several options displayed as outputs in grey, indicate that the AI has a lower confidence level, based on the features that have been selected. https://mycomputerbrain.net/php/experiments/ai.experiment26a.php

   

   Image credit: MyComputer brain (animal classification by an ANN)

Summary

Computer vision has come a long way but it is still a challenge for computer scientists to improve and build accurate systems. Currently, computer scientists teach computers in a similar way to the way teachers teach students at school. Artificial intelligences use the methods that you have seen in this lesson plan to form an understanding of the object – and this understanding is based on a picture of the object/thing, using its colour, shape, and features. AIs then combine these insights to form a judgement, a classification. Like humans, AIs are confident or not so confident. But a good AI always tells us how confident it is about its judgement. It won’t pretend to be confident if it isn’t.

Science focus

• Features of an animal are used to classify it according to class, eg mammals, birds, fish, reptiles and amphibians, but what other attributes are also used? (Examples: egg laying or live young, body temperature control and how it breathes – via gills, lungs or skin.)

Digital Technologies focus

• What data about animals can you represent? In what different ways might that data be represented, depending on the purpose of the investigation?
• What data does a computer use to see?
• What are the challenges to an AI’s accurate recognition of an object?
• In what cases might it be useful for an AI to use image recognition?

Mathematics focus

• What geometric shapes did you use to represent an animal?
• How can you make different geometric shapes by combining and cutting shapes?

In Digital Technologies, representing data refers to the way data is symbolised, visually treated or provided in audio. For students in years 3–4, the focus for data representation is on how the same data can be represented in different ways. An example of this is data in a table represented as text or as images or a combination of these.

Image recognition is an area of AI that has many applications. For a computer to recognise what it sees, it needs input of data through a camera and some form of processing. Classification is a supervised learning technique used to group data based on attributes or features. Humans can provide labels on the data input (for images or text, for example) that tell the machine both about the attributes or features (such as colour, size, shape, measurements) of the data input and how to group it. The machine then matches future data based on the similarity of the new data to predefined groups. An example: sorting images of kangaroos and wombats based on their number of legs, whether they have a tail and whether they have external ears.

• AI sees shapes (.pptx)
• Animal Classification for Children: Classifying Vertebrates and Invertebrates for Kids – FreeSchool
• Animal features classification spreadsheet
• Copyright free animal images
• Vertebrate Classes
• What is Artificial Intelligence (or Machine Learning)?
• What Is Machine Learning and How Does It Work? | Machine Learning Explained