Experiment with Vision Sensing Using fitlet2 and Pixy2
Posted: Mon Aug 26, 2019 12:26 am
Overview
Vision Sensing represents by far the best overall method of perceiving the surrounding environment. While other sensory technologies such as millimeter wave radar, lidar, acoustic, thermal, chemical , and so on are very useful, none of them provides the level, quality, and variety of information that vision does.
Previously, electro-optical sensory systems were quite expensive and were mostly found in military hardware such as the AGM-65 Maverick missile and similar systems deployed around the world. Now however, affordable and capable vision sensing devices are available to the rest of us. One such device is the Pixy2 camera which is capable of line following and object recognition. Available for around $60 US, Pixy2 is a great addition to small, capable platforms such as fitlet2.
Older Machine Vision Sensing Technology - Expensive and Limited to Military Applications
Pixy2 - Modern Machine Vision Sensing Technology - Affordable and Available to All
About Pixy2
Pixy2 is the latest incarnation of Pixy which was born of a partnership between the Carnegie Mellon Robotics Institute and Charmed Labs. The hardware and software utilized in Pixy2 are open source, so it is an ideal device for use by the makers among us.
Pixy2 has a couple of sensing modes that are quite useful for the maker - especially for robotics applications. These modes are:
1. Object Detection and Recognition
Pixy2 learns and detects objects based on colors using an algorithm known as Color Connected Components. Color based object recognition was utilized because of its efficiency, speed, and relatively "bullet-proof" performance. Pixy2 has the ability to simultaneously detect and track a large number of objects simultaneously. This is a capability that was once reserved for only very complex and expensive military systems.
2. Line Tracking
Pixy2 has the ability to track and follow lines using its custom line tracking algorithm. Unlike photodetector based solutions, Pixy2's vision based line tracking solution has the ability to anticipate by looking ahead and the ability to easily identify and navigate through intersections.
Hardware Used
The hardware used for this demonstration is:
1. fitlet2
2. Pixy2 camera system
3. Pixy2 Pan-Tilt kit
Software
The basic software package for configuring and experimenting with Pixy2 is PixyMon. It is available for Linux, Mac, and Windows. This app note focuses on the Linux version. The preferred distro for the Linux version of PixyMon is Linux Mint which is what I use.
Software Installation
Detailed instructions for installing PixyMon in Linux are found here:
https://docs.pixycam.com/wiki/doku.php? ... n_on_linux
In my case I installed PixyMon in /usr/local/src/pixy2 in order to adhere to what I perceive to be standard Linux practice - although there are many different opinions concerning what is standard practice in Linux.
Also, I wrote a short script PixyMon.sh which I made executable and placed in /home/[username]/bin for universal access.
PixyMon.sh
#!/bin/bash
# Script to Run PixyMon
cd /
./usr/local/src/pixy2/build/pixymon/PixyMon
Test Results
Pixy2 in conjunction with fitlet2 proved to be a fun and interesting system to explore. I was impressed with Pixy2's ability to accurately and consistently detect, identify, and track objects. The following images and videos provide more detail.
1. Object Detection, Recognition, and Tracking
Basic Shape/Color Recognition - Primary Colors Basic Shape/Color Recognition - Blended Secondary Colors [/size]
Basic Shape/Color Recognition - Multiple Objects Basic Shape/Color Tracking - PixyMon - Ripe Tomato
Basic Shape/Color Tracking - Pixy2 Pan/Tilt Kit - Ripe Tomato
2. Line Tracking
Since I am using Pixy2 in a more or less static configuration, I was not able to make a detailed exploration of its line tracking performance. However, the following picture shows that Pixy2 does its best to search its field of view for lines to follow.
Potential Projects
1. Lunar Tracker for aiming "Moon Bounce" laser and microwave transmissions.
2. Laser tracker for mobile platform.
3. Automated fruit/vegetable harvesting system using color/shape recognition.
4. Vehicle "road ahead" hazard detection, recognition, and warning system.
5. Etc.
Actual Line Following Project (By Others)
Personally, I think a fitlet2 would make a fine field-hardened replacement for the laptop used in the following video.
Resources for Developers, Experimentors, and Makers
APIs, Libraries, and Software
https://pixycam.com/downloads-pixy2/
Wiki
https://docs.pixycam.com/wiki/doku.php?id=wiki:v2:start
Forums
https://forum.pixycam.com/
Where Do fitlet2 and Pixy2 Go From Here?
That remains to be seen, but I bet they could be the foundation for a Very Cool Robot!!
Vision Sensing represents by far the best overall method of perceiving the surrounding environment. While other sensory technologies such as millimeter wave radar, lidar, acoustic, thermal, chemical , and so on are very useful, none of them provides the level, quality, and variety of information that vision does.
Previously, electro-optical sensory systems were quite expensive and were mostly found in military hardware such as the AGM-65 Maverick missile and similar systems deployed around the world. Now however, affordable and capable vision sensing devices are available to the rest of us. One such device is the Pixy2 camera which is capable of line following and object recognition. Available for around $60 US, Pixy2 is a great addition to small, capable platforms such as fitlet2.
Older Machine Vision Sensing Technology - Expensive and Limited to Military Applications
Pixy2 - Modern Machine Vision Sensing Technology - Affordable and Available to All
About Pixy2
Pixy2 is the latest incarnation of Pixy which was born of a partnership between the Carnegie Mellon Robotics Institute and Charmed Labs. The hardware and software utilized in Pixy2 are open source, so it is an ideal device for use by the makers among us.
Pixy2 has a couple of sensing modes that are quite useful for the maker - especially for robotics applications. These modes are:
1. Object Detection and Recognition
Pixy2 learns and detects objects based on colors using an algorithm known as Color Connected Components. Color based object recognition was utilized because of its efficiency, speed, and relatively "bullet-proof" performance. Pixy2 has the ability to simultaneously detect and track a large number of objects simultaneously. This is a capability that was once reserved for only very complex and expensive military systems.
2. Line Tracking
Pixy2 has the ability to track and follow lines using its custom line tracking algorithm. Unlike photodetector based solutions, Pixy2's vision based line tracking solution has the ability to anticipate by looking ahead and the ability to easily identify and navigate through intersections.
Hardware Used
The hardware used for this demonstration is:
1. fitlet2
2. Pixy2 camera system
3. Pixy2 Pan-Tilt kit
Software
The basic software package for configuring and experimenting with Pixy2 is PixyMon. It is available for Linux, Mac, and Windows. This app note focuses on the Linux version. The preferred distro for the Linux version of PixyMon is Linux Mint which is what I use.
Software Installation
Detailed instructions for installing PixyMon in Linux are found here:
https://docs.pixycam.com/wiki/doku.php? ... n_on_linux
In my case I installed PixyMon in /usr/local/src/pixy2 in order to adhere to what I perceive to be standard Linux practice - although there are many different opinions concerning what is standard practice in Linux.
Also, I wrote a short script PixyMon.sh which I made executable and placed in /home/[username]/bin for universal access.
PixyMon.sh
#!/bin/bash
# Script to Run PixyMon
cd /
./usr/local/src/pixy2/build/pixymon/PixyMon
Test Results
Pixy2 in conjunction with fitlet2 proved to be a fun and interesting system to explore. I was impressed with Pixy2's ability to accurately and consistently detect, identify, and track objects. The following images and videos provide more detail.
1. Object Detection, Recognition, and Tracking
Basic Shape/Color Recognition - Primary Colors Basic Shape/Color Recognition - Blended Secondary Colors [/size]
Basic Shape/Color Recognition - Multiple Objects Basic Shape/Color Tracking - PixyMon - Ripe Tomato
Basic Shape/Color Tracking - Pixy2 Pan/Tilt Kit - Ripe Tomato
2. Line Tracking
Since I am using Pixy2 in a more or less static configuration, I was not able to make a detailed exploration of its line tracking performance. However, the following picture shows that Pixy2 does its best to search its field of view for lines to follow.
Potential Projects
1. Lunar Tracker for aiming "Moon Bounce" laser and microwave transmissions.
2. Laser tracker for mobile platform.
3. Automated fruit/vegetable harvesting system using color/shape recognition.
4. Vehicle "road ahead" hazard detection, recognition, and warning system.
5. Etc.
Actual Line Following Project (By Others)
Personally, I think a fitlet2 would make a fine field-hardened replacement for the laptop used in the following video.
Resources for Developers, Experimentors, and Makers
APIs, Libraries, and Software
https://pixycam.com/downloads-pixy2/
Wiki
https://docs.pixycam.com/wiki/doku.php?id=wiki:v2:start
Forums
https://forum.pixycam.com/
Where Do fitlet2 and Pixy2 Go From Here?
That remains to be seen, but I bet they could be the foundation for a Very Cool Robot!!