Experiment with Vision Sensing Using fitlet2 and Pixy2

Moderator: hassellbear

Post Reply
Posts: 106
Joined: Mon May 28, 2012 12:25 pm

Experiment with Vision Sensing Using fitlet2 and Pixy2

Post by hassellbear »

Vision Sensing.jpg
Vision Sensing.jpg (8.84 KiB) Viewed 4857 times
Vision Sensing represents by far the best overall method of perceiving the surrounding environment. While other sensory technologies such as millimeter wave radar, lidar, acoustic, thermal, chemical , and so on are very useful, none of them provides the level, quality, and variety of information that vision does.

Previously, electro-optical sensory systems were quite expensive and were mostly found in military hardware such as the AGM-65 Maverick missile and similar systems deployed around the world. Now however, affordable and capable vision sensing devices are available to the rest of us. One such device is the Pixy2 camera which is capable of line following and object recognition. Available for around $60 US, Pixy2 is a great addition to small, capable platforms such as fitlet2.

Older Machine Vision Sensing Technology - Expensive and Limited to Military Applications
Maverick.jpg (30.48 KiB) Viewed 4857 times

Pixy2 - Modern Machine Vision Sensing Technology - Affordable and Available to All
Pixy2.jpg (46.29 KiB) Viewed 4857 times

About Pixy2

Pixy2 is the latest incarnation of Pixy which was born of a partnership between the Carnegie Mellon Robotics Institute and Charmed Labs. The hardware and software utilized in Pixy2 are open source, so it is an ideal device for use by the makers among us.
Pixy2 Open Source.jpg
Pixy2 Open Source.jpg (8.58 KiB) Viewed 4854 times

Pixy2 has a couple of sensing modes that are quite useful for the maker - especially for robotics applications. These modes are:

1. Object Detection and Recognition

Pixy2 learns and detects objects based on colors using an algorithm known as Color Connected Components. Color based object recognition was utilized because of its efficiency, speed, and relatively "bullet-proof" performance. Pixy2 has the ability to simultaneously detect and track a large number of objects simultaneously. This is a capability that was once reserved for only very complex and expensive military systems.
Orange.png (126.91 KiB) Viewed 4857 times
2. Line Tracking

Pixy2 has the ability to track and follow lines using its custom line tracking algorithm. Unlike photodetector based solutions, Pixy2's vision based line tracking solution has the ability to anticipate by looking ahead and the ability to easily identify and navigate through intersections.
Line Tracking 1.png
Line Tracking 1.png (42.06 KiB) Viewed 4857 times

Hardware Used
Hardware.jpg (26.33 KiB) Viewed 4857 times
The hardware used for this demonstration is:

1. fitlet2
2. Pixy2 camera system
3. Pixy2 Pan-Tilt kit


The basic software package for configuring and experimenting with Pixy2 is PixyMon. It is available for Linux, Mac, and Windows. This app note focuses on the Linux version. The preferred distro for the Linux version of PixyMon is Linux Mint which is what I use.
LM Friendly.jpg
LM Friendly.jpg (6.85 KiB) Viewed 4854 times
Software Installation

Detailed instructions for installing PixyMon in Linux are found here:

https://docs.pixycam.com/wiki/doku.php? ... n_on_linux

In my case I installed PixyMon in /usr/local/src/pixy2 in order to adhere to what I perceive to be standard Linux practice - although there are many different opinions concerning what is standard practice in Linux.

Also, I wrote a short script PixyMon.sh which I made executable and placed in /home/[username]/bin for universal access.

# Script to Run PixyMon
cd /

Test Results

Pixy2 in conjunction with fitlet2 proved to be a fun and interesting system to explore. I was impressed with Pixy2's ability to accurately and consistently detect, identify, and track objects. The following images and videos provide more detail.

1. Object Detection, Recognition, and Tracking

Basic Shape/Color Recognition - Primary Colors
Primary.jpg (143.77 KiB) Viewed 4854 times
Basic Shape/Color Recognition - Blended Secondary Colors
Secondary.jpg (95.87 KiB) Viewed 4854 times

Basic Shape/Color Recognition - Multiple Objects
Multi.jpeg (150.4 KiB) Viewed 4854 times
Basic Shape/Color Tracking - PixyMon - Ripe Tomato

Basic Shape/Color Tracking - Pixy2 Pan/Tilt Kit - Ripe Tomato

2. Line Tracking

Since I am using Pixy2 in a more or less static configuration, I was not able to make a detailed exploration of its line tracking performance. However, the following picture shows that Pixy2 does its best to search its field of view for lines to follow.
Line Tracking.png
Line Tracking.png (65.52 KiB) Viewed 4854 times

Potential Projects

1. Lunar Tracker for aiming "Moon Bounce" laser and microwave transmissions.
2. Laser tracker for mobile platform.
3. Automated fruit/vegetable harvesting system using color/shape recognition.
4. Vehicle "road ahead" hazard detection, recognition, and warning system.
5. Etc.

Actual Line Following Project (By Others)

Personally, I think a fitlet2 would make a fine field-hardened replacement for the laptop used in the following video.

Resources for Developers, Experimentors, and Makers

APIs, Libraries, and Software



Where Do fitlet2 and Pixy2 Go From Here?
Where To.jpeg
Where To.jpeg (4.17 KiB) Viewed 4854 times
That remains to be seen, but I bet they could be the foundation for a Very Cool Robot!!

Site Admin
Posts: 419
Joined: Mon Dec 25, 2017 4:21 pm

Re: Experiment with Vision Sensing Using fitlet2 and Pixy2

Post by tamir »

Very interesting stuff right there.
Thank you for this enriching post.

Post Reply

Return to “Cool stuff with fitlet2”