RoboCV Workshop

and
(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa
| Ajusal Sugathan & Utkarsh Sinha

View Slide

presents

View Slide


View Slide

The word robot originally was supposed to
mean a slave
It is a machine which performs a variety
of tasks, either using manual external
control or intelligent automation
A manually controlled car or a ASIMOV
trying to kick a football are all robots
(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa | Ajusal Sugathan & Utkarsh Sinha

View Slide

ž Robotics is a multi disciplinary field of
engineering encompassing the vistas of
› Mechanical design
› Electronic control
› Artificial Intelligence
ž It finds it’s uses in all aspects of our life
› automated vacuum cleaner
› Exploring the ‘Red’ planet
› Setting up a human colony there :D

View Slide

ROBOTS
CONTROL
AUTONOMOUS
MANUAL
APPLICATIONS
INDUSTRIAL
MEDICAL
INTERFACE
HARDWARE
SOFTWARE
INTERLINKED

View Slide

Ø Locomotion System
Ø Actuators
Ø Power Supply System
Ø Transmission System
Ø Switches
Ø Sensory Devices For Feedback
Ø Sensor Data Processing Unit

View Slide

Ø A mobile robot must have a system to
make it move. Ob.
Ø This system gives our machine the ability
to move forward, backward and take
turns
Ø It may also provide for climbing up and
down
Ø Or even flying or floating J

View Slide

Ø Each type of locomotion requires
different number of degrees of freedom
Ø More degrees of freedom means more the
number of actuators you will have to use
Ø Although one actuator can be used to
control more than one degree of freedom

View Slide

Ø Wheeled
Ø Legged
Ø Climbing
Ø Flying
Ø Floating
Ø Snake-Like

View Slide

Ø The kind of locomotion most frequently
used in robotics at the undergrad level
Ø This involves conversion of electrical
energy into mechanical energy (mostly
using motors)
Ø The issue is to control these motors to
give the required speed and torque

View Slide

Ø We have a simple equation for the
constant power delivered to the motor:
› P = ζ X ω
Ø Note that the torque and angular velocity
are inversely proportionally to each other
Ø So to increase the speed we have to
reduce the torque

View Slide

Ø The dc motors available have very high
speed of rotation which is generally not
needed
Ø At high speeds, they lack torque
Ø For reduction in speed and increase in
“pulling capacity” we use pulley or gear
systems

View Slide

Ø Differential Drive
Ø Dual Differential Drive
Ø Car-type Drive
Ø Skid-steer Drive
Ø Synchronous Drive

View Slide

Ø Simplest, easiest to implement and most
widely used.
Ø It has a free moving wheel in the front
accompanied with a left and right wheel. The
two wheels are separately powered
Ø When the wheels move in the same direction
the machine moves in that direction.
Ø Turning is achieved by making the wheels
oppose each other’s motion, thus generating
a couple

View Slide


View Slide

Ø In-place (zero turning radius) rotation is
done by turning the drive wheels at the same
rate in the opposite direction
Ø Arbitrary motion paths can be implemented
by dynamically modifying the angular
velocity and/or direction of the drive wheels
Ø Total of two motors are required, both of
them are responsible for translation and
rotational motion

View Slide

Ø Simplicity and ease of use makes it the most
preferred system by beginners
Ø Independent drives makes it difficult for
straight line motion. The differences in
motors and frictional profile of the two
wheels cause them to move with slight
turning effect
Ø The above drawback must be countered with
appropriate feedback system. Suitable for
human controlled remote robots

View Slide

Ø Uses synchronous rotation of its wheels to
achieve motion & turns
Ø It is made up of a system of 2 motors.
One which drive the wheels and the other
turns the wheels in a synchronous fashion
Ø The two can be directly mechanically
coupled as they always move in the same
direction with same speed

View Slide

The direction of motion is given by
black arrow. The alignment of the
machine is shown by red arrow

View Slide

Ø The use of separate motors for translation
and wheel turning guarantees straight
line motion without the need for dynamic
feedback control
Ø This system is somewhat complex in
designing but further use is much simpler

View Slide

Ø Actuators, also known as drives, are
mechanisms for getting robots to move.
Ø Most actuators are powered by
pneumatics
(air pressure), hydraulics (fluid pressure),
or motors (electric current).
Ø They are devices which transform an
input signal (mainly an electrical signal))
into motion

View Slide

Ø Widely used because of their
small size and high energy output.
Ø Operating voltage: usually 6,12,24V.
Ø Speed: 1-20,000 rpm..
Ø Power: P = ζ X ω

View Slide

ØThe stator is the
stationary outside part of a
motor.
Ø The rotor is the inner
part which rotates.
Ø Red represents a
magnet or winding with a
north polarization.
Ø Green represents a
magnet or winding with a
south polarization.
Ø Opposite, red and
green, polarities attract.
Ø Commutator contacts
are brown and the brushes
are dark grey.

View Slide

Ø Stator is composed of two or more
permanent magnet pole pieces.
Ø Rotor composed of windings which are
connected to a mechanical commutator.
Ø The opposite polarities of the energized
winding and the stator magnet attract and the
rotor will rotate until it is aligned with the
stator.
Ø Just as the rotor reaches alignment, the
brushes move across the commutator contacts
and energize the next winding.
Ø A yellow spark shows when the brushes
switch to the next winding.

View Slide

ØIt is an electric
motor that can divide a
full rotation into a large
number of steps.
Ø The motor's position
can be controlled
precisely, without any
feedback mechanism.
Ø There are three
types:
Ø Permanent
Magnet
Ø Variable
Resistance
Ø Hybrid type

View Slide

Ø Stepper motors work in a similar way to
dc motors, but where dc motors have 1
electromagnetic coil to produce movement,
stepper motors contain many.
Ø Stepper motors are controlled by
turning each coil on and off in a sequence.
Ø Every time a new coil is energized, the
motor rotates a few degrees, called the
step angle.

View Slide


View Slide

Full Step
Ø Stepper motors have 200
rotor teeth, or 200 full steps
per revolution of the motor
shaft.
Ø Dividing the 200 steps
into the 360º's rotation
equals a 1.8º full step angle.
Ø Achieved by energizing
both windings while
reversing the current
alternately.

View Slide

ØServos operate on the principle of negative
feedback, where the control input is compared to
the actual position of the mechanical system as
measured.
ØAny difference between the actual and wanted
values (an "error signal") is amplified and used to
drive the system in the direction necessary to
reduce or eliminate the error
ØTheir precision movement makes them ideal for
powering legs, controlling rack and pinion steering,
to move a sensor around etc.

View Slide

Ø Suitable power source is needed to run
the robots
Ø Mobile robots are most suitably powered
by batteries
Ø The weight and energy capacity of the
batteries may become the determinative
factor of its performance

View Slide

Ø For a manually controlled robot, you can
use batteries or voltage eliminators
(convert the normal 220V supply to the
required DC voltage 12V , 24V etc.)

View Slide

Ø Gear
Ø Belt Pulley
Ø Chain Sprocket
Ø Rack and Pinion
Ø Pick Place Mechanisms

View Slide

Ø Gears are the most common means of
transmitting power in mechanical
engineering
Ø Gears form vital elements of mechanisms in
many machines such as vehicles, metal
tooling machine tools, rolling mills, hoisting
etc.
Ø In robotics its vital to control actuator speeds
and in exercising different degrees of
freedom

View Slide


View Slide

Ø To achieve torque magnification and
speed reduction
Ø They are analogous to transformers in
electrical systems
Ø It follows the basic equation:
Ø ω1
x r1
= ω2
x r2
Ø Gears are very useful in transferring
motion between different dimension

View Slide

Ø An arrangement of gears to convert
rotational torque to linear motion
Ø Same mechanism used to steer wheels
using a steering
Ø In robotics used extensively in clamping
systems

View Slide


View Slide

Ø It allows for mechanical power, torque,
and speed to be transmitted across axes
Ø If the pulleys are of differing diameters, it
gives a mechanical advantage
Ø In robotics it can be used in lifting loads
or speed reduction
Ø Also it can be used in a differential drive
to interconnect wheels

View Slide


View Slide

Ø Sprocket is a profiled wheel with teeth that
meshes with a chain
Ø It is similar to the system found in bicycles
Ø It can transfer rotary motion between shafts
in cases where gears are unsuitable
Ø Can be used over a larger distance
Ø Compared to pulleys has lesser slippage due
to firm meshing between the chain and
sprocket

View Slide


View Slide

Ø For picking and placing many mechanisms
can be used:
vHook and pick
vClamp and pick
vSlide a sheet below and pick
vMany other ways
vLots of Scope for innovation

View Slide


View Slide

ž Image Processing is a tool for
analyzing image data in all areas of
natural science
ž It is concerned with extracting data
from real-world images
ž Differences from computer graphics is
that computer graphics makes
extensive use of primitives like lines,
triangles & points. However no such
primitives exist in a real world images.

View Slide

ž Increasing need to replicate human
sensory organs
ž Eye (Vision) : The most useful and
complex sensory organ

View Slide


View Slide

ž Automated visual inspection system
Checking of objects for defects visually
ž Remote Sensing
ž Satellite Image Processing
ž Classification (OCR), identification
(Handwriting, finger prints) etc.
ž Detection and Recognition systems
(Facial recognition..etc)
ž Biomedical applications

View Slide

ž Camera, Scanner or any other image
acquisition device
ž PC or Workstation or Digital Signal
Processor for processing
ž Software to run on the hardware platform
(Matlab, Open CV etc.)
ž Image representation to process the
image (usually matrix) and provide spatial
relationship
ž A particular color space is used to
represent the image

View Slide

Image Acquisition Device
(Eg. CCD or CMOS Camera)
Image Processor
(Eg. PC or DSP)
Image Analysis Tool
(Eg. Matlab or Open CV)
Machine Control Of Hardware
through serial or parallel interfacing

View Slide

ž Using a camera
ž Analog cameras
ž Digital cameras
› CCD and CMOS cameras
ž Captures data from a single
light receptor at a time
ž CCD – Charge Coupled Devices
ž CMOS – Complementary MOSFET Sensor
based

View Slide

ž Digital Cameras
› CCD Cameras
– High quality, low noise images
– Genarates analog signal converted using ADC
– Consumes high power
› CMOS Cameras
– Lesser sensitivity
– Poor image quality
– Lesser power
ž Analogue cameras require grabbing card
or TV tuner card to interface with a PC

View Slide

Colored pixels on CCD
Chip

View Slide

ž Matlab
ž Open CV

View Slide


View Slide

ž Two types: Vector and Raster
ž Vector images store curve information
ž Example: India’s flag
ž Three rectangles, one circle and
the spokes
ž We will not deal with vector images at
all

View Slide

ž Raster images are different
ž They are made up of several dots

View Slide

ž If you think about it, your laptop’s
display is a raster display
ž Also, vector images are high level
abstractions
ž Vector representations are more
complex and used for specific purposes

View Slide

ž Raster
› Matrix
ž Vector
› Quadtrees
› Chains
› Pyramid
Of the four, matrix is the most general.
The other three are used for special
purposes. All these representations must
provide for spatial relationships

View Slide

ž Computers cannot handle continuous
images but only arrays of digital
numbers
ž So images are represented as 2-D
arrays of points (2-D matrix)(Raster
Represenatation)
ž A point on this 2-D grid (corresponding
to the image matrix element) is called
PIXEL (picture element)
ž It represents the average irradiance
over the area of the pixel

View Slide

ž Each pixel requires some memory
ž Color depth : Amount of memory each
pixel requires
ž Examples
› 1-bit
› 8-bit
› 32-bit
› 64-bit

View Slide

ž Pixels are tiny little dots of color you see on
your screen, and the smallest possible size
any image can get
ž When an image is stored, the image file
contains information on every single pixel in
that image i.e
› Pixel Location
› Intensity
ž The number of pixels used to represent the
image digitally is called Resolution
ž More the number of pixels used, higher the
resolution
ž Higher resolution requires more
processing power

View Slide


View Slide

ž MATLAB stands for MATrix LABoratory,
a software developed by Mathworks
Inc (www.mathworks.com). MATLAB
provides extensive library support for
various domains of scientific and
engineering computations and
simulations

View Slide

ž When you click the MATLAB icon (from
your desktop or Start>All Programs),
you typically see three windows:
Command Window, Workspace and
Command History. Snapshots of these
windows are shown below

View Slide

ž This window shows the variables
defined by you in current session on
MATLAB

View Slide

ž Command History stores the list of
recently used commands for quick
reference

View Slide

ž This is where you run your code

View Slide

ž In MATLAB, variables are stored as
matrices (singular: matrix), which
could be either an integer, real
numbers or even complex numbers
ž These matrices bear some
resemblance to array data structures
(used in computer programming)

View Slide

ž Let us start with writing simple
instructions on MATLAB command
window
ž To define an integer,
ž Type a=4 and hit enter
ž >>a=4
ž To avoid seeing the variable, add a
semicolon after the instruction
ž >>a=4;

View Slide

ž Similarly to define a 2x2 matrix, the
instruction in MATLAB is written as
ž >> b=[ 1 2; 3 4];
ž If you are familiar with operations on
matrix, you can find the determinant
or the inverse of the matrix.
ž >> determin= det(b)
ž >> d=inv(b)

View Slide

ž Images as we have already seen are
stored as matrices
ž So now we try to see this for real on
MATLAB
ž We shall also look into the basic
commands provided by MATLAB’s
Image Processing Toolbox

View Slide

ž Once you have started MATLAB, type the
following in the Command Window
ž >> im=imread(‘sample.jpg');
ž This command stores the file image file
‘sample.jpg’ in a variable called ‘im’
ž It takes this file from the Current-
Directory specified
ž Else, entire path of file should be
mentioned

View Slide

ž You can display the image in another
window by using imshow command
ž >>figure,imshow(im);
ž This pops up another window (called as
figure window), and displays the image
‘im’

View Slide

ž The ‘imview’ command can also be
used in order toview the image
ž imview(im);
ž Difference is that in this case you can
see specific pixel values just by moving
the cursor over the image

View Slide

ž To know the breadth and height of the
image, use the size function,
ž >>s=size(im);
ž The size function basically gives the
size of any array in MATLAB
ž Here we get the size of the IMAGE
ARRAY

View Slide

ž Now that we have our image stored in
a variable we can observe and
understand the following:
ž How pixels are stored?
ž What does the values given by each
pixel indicate?
ž What is Image Resolution?

View Slide

ž Have a look at the values stored
ž Say the first block of 10 x 10
ž >>im(1:10,1:10);
ž Or Say view the pixel range 50:150 on
both axis
ž >>
figure,imshow(im(50:150,50:150));

View Slide


View Slide

ž 1-bit = BLACK or WHITE
ž 8-bit = 28 different shades
ž 24-bit = 224 different shades
ž 64-bit images – High end displays
ž Used in HDRI, storing extra
information per pixel, etc

View Slide

ž This is another name for 1-bit images
ž Each pixel is either White or Black
ž Technically, this is a black & white
image

View Slide


View Slide

ž Another name for 8-bit images
ž Each pixel can be one of 256 different
shades of gray
ž These images are popularly called
Black & White. Though, this is
technically wrong.

View Slide


View Slide

ž Again, each pixel gets 8 bits
ž But each of the 256 values maps to a
color in a predefined “palette”
ž If required, you can have different
bit depths

View Slide

ž We won’t be dealing with indexed
images

View Slide

ž 8-bits is too less for all the different
shades of colors we see
ž So 24-bits is generally used for color
images
ž Thus each pixel can have one of 224
unique colors

View Slide


View Slide

ž Now, a new problem arises:
ž How do you manage so many different
shades?
ž Programmers would go nuts
ž Then came along the idea of color
spaces

View Slide


View Slide

ž A color space can be thought of as a
way to manage millions of colors
ž Eliminates memorization, and
increases predictability
ž Common color spaces:
› RGB
› HSV
› YCrCb or YUV
› YIQ

View Slide

ž You’ve probably used this already

View Slide

ž Each pixel stores 3 bytes of data
ž The 24-bits are divided into three 8-bit
values
ž The three are: Red, Green and Blue i.e
the primary colours
ž Mixing of primary colours in right
proportions gives any particular colour
ž Each pixel has these 3 values

View Slide

ž 1 byte = 8 bits can store a value
between 0-255
ž We get pixel data in the form RGB
values with each varying from 0-255
ž That is how displays work
ž So there are 3 grayscale channels

View Slide


View Slide

ž Advantages:
› Intuitive
› Very widely used
ž Disadvantages:
› Image processing is relatively tough

View Slide

ž HSV makes image processing easier
ž Again, 24 bits = three 8-bit values or 3
channels
ž The 3 channels are:
› Hue
› Saturation (Shade of Colour)
› Value (Intensity)

View Slide

ž The Hue is the tint of color used
› It represents the colour of the pixel (Eg. Red
Green Yellow etc)
ž The Saturation is the “amount” of
that tint
› It represents the intensity of the colour (Eg.
Dark red and light red)
ž The Value is the “intensity” of that
pixel
› It represents the intensity of brightness of the
colour

View Slide

ž RGB image converted to HSV
RGB
HUE
SATURATION
VALUE

View Slide

ž Advantages:
› The color at a pixel depends on a single
value
› Illumination independent
ž Disadvantages:
› Something

View Slide

ž Intuitively RGB might seem to be the simpler
and better colour space to deal with
ž Though HSV has its own advantages
especially in colour thresholding
ž As the colour at each pixel depends on a
single hue value it is very useful in
separating out blobs of specific colours even
when there are huge light variations
ž Thus it is very useful in processing real
images taken from camera as there is a large
amount of intensity variation in this case
ž Hence, ideal for robotics applications

View Slide

ž Widely used in digital video
ž Has three 8-bit channels:
› Y Component:
– Gives luminance or intensity
› Cr Component:
– It is the RED component minus a reference value
› Cb Component:
– It is the BLUE component minus a reference
value
ž Hence Cr and Cb components represent
the colour called “Color Difference
Components”

View Slide

ž Advantages:
› Used in video processing
› Gives you a 2-D colour space hence helps
in closer distinguishing of colours
ž Disadvantages:

View Slide


View Slide

ž The camera returns images in a certain
color space
ž You might want to convert to different
color spaces to process it
ž Colour space conversions can take
place between RGB to any other colour
space and vice versa

View Slide

ž Since cameras usually input images in
rgb
ž We would like to convert these images
into HSV or YCrCb
ž Conversions:
› RGB->HSV
› HSV->RGB
› RGB->YCrCb
› YCrCb->RGB

View Slide

ž RGB -> HSV

View Slide

(c) 2009-2010 Electronics & Robotics Club, BITS-Pilani, Goa |
Ajusal Sugathan & Utkarsh Sinha
ž HSV RGB YCrCb

View Slide

ž >>h = rgb2hsv(im)
ž This converts the RGB image to HSV
ž The new colour space components can be
seen using
ž >> imview(h)
ž >> imview(h(:,:,1)) “—HUE—”
ž >> imview(h(:,:,2)) “—Saturation—
”
ž >> imview(h(:,:,3)) “—Value—”

View Slide

ž >>R = hsv2rgb(im)
ž This converts the HSV image to RGB
ž The new colour space components can
be seen using
ž >> imview(R)
ž >> imview(R(:,:,1)) “—Red—”
ž >> imview(R(:,:,2)) “—Green—”
ž >> imview(R(:,:,3)) “—Blue—”

View Slide

ž >> Y = rgb2ycbcr(im);
ž This converts the RGB image to YCbCr
ž The new colour space components can be
seen using
ž >> imview(Y)
ž >> imview(Y(:,:,1)) “—Luminance—”
ž >> imview(Y(:,:,2)) “—Differenced
Blue—”
ž >> imview(Y(:,:,3)) “—Differenced
Red—”

View Slide

ž >> R = ycbcr2rgb(im);
ž This converts the YCbCr image to RGB
ž The new colour space components can
be seen using
ž >> imview(R)
ž >> imview(R(:,:,1)) “—Red—”
ž >> imview(R(:,:,2)) “—Green—”
ž >> imview(R(:,:,3)) “—Blue—”

View Slide

ž Formulae for conversion are very
complex
ž But the best thing is, you don’t need to
remember these formulae
ž Matlab and OpenCV have built-in
functions for these transformations :-)

View Slide


View Slide

ž OpenCV is a collection of many
functions that help in image processing
ž You can use OpenCV in C/C++, .net
languages, Java, Python, etc as well
ž We will only discuss OpenCV in C/C++

View Slide

ž It is blazingly fast
ž Quite simple to use and learn
ž Has functions for machine learning,
image processing, and GUI creation

View Slide


View Slide

ž Download the latest OpenCV package
from
http://sourceforge.net/projects/opencv
/
ž Install the package, and note where
you installed it (like C:\Program
Files\OpenCV\)

View Slide

ž Now, we need to tell Microsoft Visual
Studio that we’ve installed OpenCV
ž So, we tell it where to find the OpenCV
header files
ž Start Microsoft Visual Studio 2008

View Slide


View Slide

1
2

View Slide

Type
these
paths
into the
list

View Slide

ž Right now, Visual Studio knows where
to find the OpenCV include files and
library files
ž Now we create a new project

View Slide


View Slide

ž Accept all default settings in the
project
ž You’ll end up with an empty project
with a single file (like Mybot.cpp)
ž Open this file, we’ll write some code
now

View Slide

ž Add the following at the top of the
code
#include
#include
ž This piece of code includes necessary
OpenCV functionality

View Slide

ž Now, we get to the main() function
int main()
{
ž The main function is where for
program execution begins

View Slide

ž Next, we load an image
IplImage* img = cvLoadImage("C:\\hello.jpg");
ž The IplImage is a data type, like int,
char, etc

View Slide

ž Comes built-into OpenCV
ž Any image in OpenCV is stored as an
IplImage thingy
ž It is a “structure”

View Slide

ž Opens filename and returns it as an
IplImage structure
ž Supported formats:
› Windows bitmaps - BMP, DIB
› JPEG files - JPEG, JPG, JPE
› Portable Network Graphics - PNG
› Portable image format - PBM, PGM, PPM
› Sun rasters - SR, RAS
› TIFF files - TIFF, TIF
› OpenEXR HDR images - EXR
› JPEG 2000 images - jp2

View Slide

ž Now we show this image in a window
cvNamedWindow("myfirstwindow");
cvShowImage("myfirstwindow", img);
ž This uses some HighGUI functions
(comes along with OpenCV)

View Slide

ž Creates a window with the caption title
ž This is a HighGUI function
ž You can add controls to each window
as well (track bars, buttons, etc)

View Slide

ž Shows img in the window with caption
title
ž If no such window exists, nothing
happens

View Slide

ž Finally, we wait for an input, release
and exit
cvWaitKey(0);
cvReleaseImage(&img);
return 0;
}

View Slide

ž Waits for time milliseconds, and
returns whatever key is pressed
ž If time=0, waits till eternity
ž Here, we’ve used it to keep the
windows from vanishing immediately

View Slide

ž Erases img from the RAM
ž Get rid of an image as soon as
possible. RAM is precious J
ž Note that you send the address of the
image (&img) and not just the image
(img)

View Slide


View Slide

ž Right now, Visual Studio knows where
OpenCV is
ž But it does not know, whether to use
OpenCV or not
ž We need to tell this explicitly

View Slide


View Slide

ž Got errors?
› Check if the syntax is correct
› Copy all DLL files in *\OpenCV\bin\ into
C:\Windows\System32

View Slide


View Slide

ž src is the original image
ž dst is the destination
ž code is one of the follow:
› CV_BGR2HSV
› CV_RGB2HSV
› CV_RGB2YCrCb
› CV_HSV2RGB
› CV_2

View Slide

ž src should be a valid image. Or an
error will pop up
ž dst should be a valid image, i.e. you
need a blank image of the same size
ž code should be valid (check the
OpenCV documentation for that)

View Slide

ž Allocates memory for an image of size
size, with bits bits/pixel and chan
number of channels
ž Used for creating a blank image
ž Use cvSize(width, height) to specify
the size

View Slide

ž Example:
› IplImage* blankImg =
cvCreateImage(cvSize(640, 480), 8, 3);

View Slide


View Slide

ž Wired
› Motor Driving module
› Interface with PC (Parallel/Serial)
ž Wireless
› The Motor-driving module
› The Wireless Receiver Circuit
› The Wireless Transmitter Circuit
› Interface with PC (Parallel/Serial)

View Slide


View Slide

ž IC 7805 Voltage Regulator
ž L293D Motor Driver
ž MCT2E Opto-Coupler
ž Parallel Port Male-Connector
ž RF-RX Connector

View Slide

ž It’s a three terminal linear 5 volt
regulator used to supply the board and
other peripherals
ž Prescribed input voltage to this
component is about 7-9 Volts

View Slide

ž Voltage fluctuations can be controlled
by using low pass filter capacitors
across output and input
ž Higher input voltage can be applied if
heatsink is provided

View Slide

ž Used to control Dc and Stepper Motors
ž Uses a H-Bridge which is an electronic
switching circuit that can reverse direction of
current
ž It’s a Dual-H bridge
ž Basically used to convert a low voltage input
into a high voltage output to drive the motor
or any other component
ž Eg: Microcontrollerà Motor Driverà
Motor
(5 Volts) (12
Volts)

View Slide

ž Different Motor Driver ICs
› L293D
– 600mA Current Rating
– Dual H-bridge (Dc and Stepper Motors)
› L298N
– 1 Amp Current Rating
› L297-L298 (Coupled)
– For stepper motor overdriving
– 2 Ics in parallel
› ULN2003/ULN2803
– 500mA Current Rating
– For unipolar stepper motors

View Slide


View Slide

ž Output Current:
› 600 mA
ž Output Voltage
› Wide Range
› 4.5 V – 36 V

View Slide


View Slide

ž There are many situations where
signals and data need to be transferred
from one subsystem to another within
a piece of electronics
ž Relays are too bulky as they are
electromechanical in nature and at the
same time give lesser efficiency
ž In these cases an electronic
component called Optocoupler is used

View Slide

ž They are generally used when the 2
subsystems are at largely different
voltages
ž These use a beam of light to transmit
the signals or data across an electrical
barrier, and achieve excellent isolation

View Slide

ž In our circuit, Opto-isolator (MCT2E) is
used to ensure electrical isolation
between motors and the PC parallel port
during wired connection
ž The Viz-Board has four such chips to
isolate the four data lines (pin 2, pin 3,
pin 4, pin 5) coming out of the parallel
port

View Slide


View Slide

ž Along with the Viz-Board 2 extensions
have been provided i.e
› The Rf Transmitter Module
› The Rf Reciever Module
Transmitt
er
Receive
r

View Slide

ž Radio frequency modules are used for
data transmission wirelessly at a certain
frequency
ž It sends and receives radio waves of a
particular frequency and a decoder and
encoder IC is provided to encode and
decode this information
ž Wireless transmission takes place at a
particular frequency Eg. 315Mhz
ž Theses modules might be single or dual
frequency

View Slide

ž Antenna is recommended on both of
them - just connect any piece of 23 cm
long to the Antenna pin
ž The kit has a dual frequency RF
module with frequencies 315/434 Mhz

View Slide


View Slide

ž The encoder IC encodes the parallel
port data and sends it to the RF
transmitter module for wireless
transmission
ž They are capable of encoding
information which consists of N
address bits and (12-N) data bits

View Slide

ž The HT12E Encoder IC has 8 address
bits and 4 data bits
ž A DIP-Switch can be used to set or
unset the address bits A0-A7

View Slide

A0-A7—Address
Bits
AD8-AD11—Data
Bits

View Slide

ž The decoder IC decodes the RF
transmitter data and sends it to the
parallel port for wireless transmission
ž They are capable of encoding
information which consists of N
address bits and (12-N) data bits

View Slide

ž The HT12D Decoder IC has 8 address
bits and 4 data bits
ž A DIP-Switch can be used to set or
unset the address bits A0-A7

View Slide

A0-A7—Address
Bits
D8-D11—Data
Bits

View Slide


View Slide

ž Serial Port
ž Parallel Port
ž USB

View Slide

ž Data is transferred serially i.e packets
are sent one after the other through a
single port

View Slide


View Slide

ž Data is transferred in parallel through
different data pins at the same time
ž Communication is pretty fast
ž Found in old printer ports

View Slide


View Slide

25th pin : Ground
2nd-12th pin : I/O lines

View Slide

ž Parallel port is faster than serial
ž A mass of data can be transmitted at
the same time through parallel ports
ž Though parallel and serial ports are not
found these days in laptops
ž Desktops and old laptops have these
ports

View Slide

Direct Output from parallel port
Output from motor driver

View Slide


View Slide

Camera, object and source positions

View Slide

Image sampling
and quantization

View Slide

Continuous image
projected
on an array sensor
Result of image sampling
and quantization

View Slide

Sampling:
Digitizing the coordinate
values
(spatial resolution)
Quantization:
Digitizing the amplitude
values
(intensity levels)

View Slide

• 1 bit /pixel
• B bits/pixel
–2B gray levels
–1 byte = 8 bits –> 256
levels
–2 possible values
–2 gray levels -> 0 or 1 (binary
image)

View Slide

ž All this sampling and quantization puts
in extra noise on the image!
ž Noise can be reduced by
› Using hardware
› Using software: filters

View Slide


View Slide

ž Why do we need to enhance images?
ž Why filter images?

View Slide

ž Large amounts of external
disturbances in real images
ž Due to different factors like changing
lighting and other real-time effects
ž To improve quality of a captured image
to make it easier to process the image

View Slide

ž First step in most IP applications
ž Used to remove noise in the input
image
ž To remove motion blur from an image
ž Enhancing the edges of an image to
make it appear sharper

View Slide

ž Generally used types Of Filtering
› Averaging Filter
› Mean Filter
› Median Filter
› Gaussian Smoothing
› Histogram Equalization

View Slide

ž The Averaging filter is used to sharpen
the images by taking average over a
number of images
ž It eliminates noise by assuming that
different snaps of the same image
have different noise patterns

View Slide

ž Noise is gaussian in nature i.e follows a
gaussian curve
ž Hence, summing up noises infinite times
approaches zero

View Slide


View Slide

ž This is extremely useful for satellites
that take intergalactic photographs
ž The images are extremely faint, and
there is more noise than the image
itself
ž Millions of pictures are taken, and
averaged to get a clear picture

View Slide

ž The Mean is used to soften an image
by averaging surrounding pixel values
Center pixel =
(22+77+48+150+77+158+0+77+219)/9

View Slide

ž The center pixel would be changed
from 77 to 92 as that is the mean
value of all surrounding pixels
ž This filter is often used to smooth
images prior to processing
ž It can be used to reduce pixel flicker
due to overhead fluorescent lights

View Slide

ž This replaces each pixel value by the
median of its neighbors, i.e. the value
such that 50% of the values in the
neighborhood are above, and 50% are
below
ž This can be difficult and costly to
implement due to the need for sorting
of the values
ž However, this method is generally very
good at preserving edges

View Slide


View Slide

ž Its performance is particularly good for
removing short noise
ž The median is calculated by first sorting all
the pixel values from the surrounding
neighborhood into numerical order and then
replacing the pixel being considered with the
middle pixel value
ž If the neighborhood under consideration
contains an even number of pixels, the
average of the two middle pixel values is
used

View Slide

ž Used to `blur' images and remove
detail and noise
ž The effect of Gaussian smoothing is to
blur an image
ž The Gaussian outputs a `weighted
average' of each pixel's neighborhood,
with the average weighted more
towards the value of the central pixels

View Slide

ž A Gaussian provides gentler smoothing
and preserves edges better than a
similarly sized mean filter
Before
Blurring
After
Blurring

View Slide

ž It is very useful in contrast enhancement
ž Especially to eliminate noise due to
changing lighting conditions etc
ž Transforms the values in an intensity
image so that the histogram of the output
image approximately matches a specified
histogram

View Slide


View Slide

Filters and histograms

View Slide

ž ‘Imfilter’ function is used for creating
different kinds of filters In MATLAB
ž B = imfilter(A,H,’option’) filters the
multidimensional array A with the
multidimensional filter H
ž The array A can be a nonsparse numeric
array of any class and dimension
ž The result B has the same size and class as A

View Slide

ž Options in imfilter
ž Convolution is same as correlation except
that the h matrix is inverted before applying
the filter

View Slide

ž h = ones(5,5) / 25;
ž imsmooth = imfilter(im,h);
ž Here a mean filter is implemented using
the appropriate ‘h’ matrix
ž imshow(im), title('Original Image');
ž figure, imshow(imsmooth),
title('Filtered Image')

View Slide


View Slide

ž FSPECIAL is used to create predefined
filters
ž h = FSPECIAL(TYPE);
ž FSPECIAL returns h as a computational
molecule, which is the appropriate
form to use with imfilter

View Slide


View Slide

ž The process of adjusting intensity
values can be done automatically by
the histeq function
ž >>im = imread('pout.tif');
ž >>jm = histeq(im);
ž >>imshow(jm)
ž >>figure, imhist(jm,64)

View Slide

Original Image

View Slide

Histogram Equalized Image

View Slide


View Slide

ž Things aren’t as simple as they were in
Matlab
ž C/C++ needs a bit of syntax and
formalities

View Slide

ž We’ll try doing the following right now
› Gaussian filter
› Median filter
› Bilateral filter
› Simple blur
› Averaging filter

View Slide

ž Start Microsoft Visual Studio 2008
ž I assume you have OpenCV installed

View Slide


View Slide

#include
#include

View Slide

#include
#include
int main()
{

View Slide

#include
#include
int main()
{
IplImage* img = cvLoadImage(“C:\\noisy.jpg”);

View Slide

#include
#include
int main()
{
IplImage* img = cvLoadImage(“C:\\noisy.jpg”);
IplImage* imgBlur = cvCreateImage(cvGetSize(img),
8, 3);
IplImage* imgGaussian = cvCreateImage(cvGetSize
(img), 8, 3);
IplImage* imgMedian = cvCreateImage(cvGetSize
(img), 8, 3);
IplImage* imgBilateral = cvCreateImage(cvGetSize
(img), 8, 3);

View Slide

cvSmooth(img, imgBlur, CV_BLUR, 3, 3);
cvSmooth(img, imgGaussian, CV_GAUSSIAN, 3, 3);
cvSmooth(img, imgMedian, CV_MEDIAN, 3, 3);
cvSmooth(img, imgBilateral, CV_BILATERAL, 3, 3);

View Slide

cvNamedWindow(“original”);
cvNamedWindow(“blur”);
cvNamedWindow(“gaussian”);
cvNamedWindow(“median”);
cvNamedWindow(“bilateral”);
cvShowImage(“original”, img);
cvShowImage(“blur”, imgBlur);
cvShowImage(“gaussian”, imgGaussian);
cvShowImage(“median”, imgMedian);
cvShowImage(“bilateral”, imgBilateral);
cvWaitKey(0);
return 0;
}

View Slide


View Slide

ž Blur: The plain simple Photoshop blur
ž Gaussian: The best result (preserved
edges and smoothed out noise)
ž Median: Nothing special
ž Bilateral: Got rid of some noise, but
preserved edges to a greater extend

View Slide


View Slide

ž Your OpenCV installation comes with
detailed documentation
ž *\OpenCV\docs\index.html
ž Scroll down, and you’ll see OpenCV
Reference Manuals

View Slide

ž Try looking up cvSmooth in the CV
Reference Manual

View Slide

ž Now try looking up cvEqualizeHist

View Slide

ž There are no built-in functions for this
ž So, we’ll code it ourselves
ž And this will be a good exercise for
getting better at OpenCV

View Slide


View Slide

#include
#include

View Slide

#include
#include
int main()
{

View Slide

#include
#include
int main()
{
IplImage* imgRed[25];
IplImage* imgGreen[25];
IplImage* imgBlue[25];
Holds the R, G and B
channels separately
for each of the 25
images

View Slide

IplImage* imgBlue[25];
for(int i=0;i<25;i++)
{
IplImage* img;
char filename[150];
sprintf(filename, "%d.jpg", (i+1));
img = cvLoadImage(filename);
• Generate the strings “1.jpg”,
“2.jpg”, etc and store them into
filename
• Load the image filename

View Slide

img = cvLoadImage(filename);
imgRed[i] = cvCreateImage(cvGetSize(img), 8,
1);
imgGreen[i] = cvCreateImage(cvGetSize(img), 8,
1);
imgBlue[i] = cvCreateImage(cvGetSize(img), 8,
1);
• Allocate memory for each component
of image i

View Slide

imgBlue[i] = cvCreateImage(cvGetSize(img), 8,
1);
cvSplit(img, imgBlue[i], imgGreen[i],
imgRed[i], NULL);
cvReleaseImage(&img);
}
• Split img into constituent channels
• Note the order: B G R
• Release img

View Slide

ž We created 75 grayscale images: 25
for red, 25 for green and 25 for blues
ž Loaded 25 color images in the loop
ž Split each image, and stored in an
appropriate grayscale image

View Slide

CvSize imgSize = cvGetSize(imgRed[0]);
IplImage* imgResultRed = cvCreateImage(imgSize, 8,
1);
IplImage* imgResultGreen = cvCreateImage(imgSize,
8, 1);
IplImage* imgResultBlue = cvCreateImage(imgSize,
8, 1);
IplImage* imgResult = cvCreateImage(imgSize, 8,
3);
• This will hold the final, filtered image
• It will be a combination of the
grayscale channels imgResultRed,
imgResultGreen and imgResultBlue

View Slide

IplImage* imgResult = cvCreateImage(imgSize, 8,
3);
for(int y=0;y{
for(int x=0;x{
• Two loops to take us through the
entire image

View Slide

for(int x=0;x{
int theSumRed=0;
int theSumGreen=0;
int theSumBlue=0;
{
• To figure out the average, we need to
find the numerator (the sum) over all
25 images

View Slide

{
theSumRed+=cvGetReal2D(imgRed[i], y,
x);
theSumGreen+=cvGetReal2D(imgGreen[i],
y, x);
theSumBlue+=cvGetReal2D(imgBlue[i], y,
x);
}
• To figure out the average, we need to
find the numerator (the sum) over all
25 images

View Slide

theSumRed = (float)theSumRed/25.0f;
theSumGreen = (float)theSumGreen/25.0f;
theSumBlue = (float)theSumBlue/25.0f;
cvSetReal2D(imgResultRed, y, x,
theSumRed);
cvSetReal2D(imgResultGreen, y, x,
theSumGreen);
cvSetReal2D(imgResultBlue, y, x,
theSumBlue);
}
}
• Once we have the sum, we divide by
25 and set the appropriate pixels

View Slide

cvMerge(imgResultBlue, imgResultGreen,
imgResultRed, NULL, imgResult);
cvNamedWindow("averaged");
cvShowImage("averaged", imgResult);
cvWaitKey(0);
return 0;
}
• Merge the three channels, and
display the image

View Slide

ž cvLoadImage always loads as BGR
ž cvSplit to get the individual channels
ž cvMerge to combine individual
channels into a color image

View Slide

ž IplImage to store any image in
OpenCV
ž cvCreateImage to allocate memory
ž cvReleaseImage to erase an image
from the RAM

View Slide

ž cvWaitKey to get a keypress within
certain milliseconds
ž cvNamedWindow to create a window
ž cvShowImage to show an image in a
window

View Slide

ž cvGetReal2D to get value at a pixel in
grayscale images
ž cvSetReal2D to set the value at a pixel
ž CvSize to store an image’s size

View Slide

ž you can always refer to the OpenCV
documentation

View Slide


View Slide

ž The process of extracting image
components that are useful in
representation of image for some
particular purpose
ž Basic morphological operations are:
› Dilation
› Erosion

View Slide

ž The operation that grows or thickens
objects in a binary image
ž The specific manner of thickening is
controlled by a shape referred to as
“structuring element”

View Slide

Structuring
Element
Binary
Image

View Slide

Dilated
Image

View Slide


View Slide

ž Erosion shrink or thins objects in a
binary image
ž The manner of shrinkage is controlled
by the structuring element

View Slide

Structuring
Element
Binary
Image

View Slide

Eroded
Image

View Slide


View Slide

ž In practical image processing dilation
and erosion are performed in various
combinations
ž An image can undergo a series for
diltions and erosion using the same or
different structuring element

View Slide

ž In practical image processing dilation
and erosion are performed in various
combinations
ž An image can undergo a series for
diltions and erosion using the same or
different structuring element
ž Two Common Kinds:
› Morphological Opening
› Morphological Closing

View Slide

ž It is basically one erosion followed by
one dilation by the same structuring
element
ž They are used to smooth object
contours, break thin connections and
remove thin protrusions

View Slide

A—Image
B—Structuring element

View Slide


View Slide

ž It is basically one dilation followed by one
erosion by the same structuring element
ž They are used to smooth object contours
like opening
ž But unlike opening they generally join
narrow breaks, fill long thin gulfs and fills
holes smaller than the structuring
element

View Slide

A—Image
B—Structuring element

View Slide


View Slide

ž Used to generate a structuring
element
ž >>se=strel(shape,parameters)

View Slide


View Slide

ž Dilation in matlab is done using the
following command:
ž >>bw2=imdilate(bw,st)
ž Bw = Original image
ž St = Structuring element

View Slide

ž Erosion in matlab is done using the
following command:
ž >>bw2=imerode(bw,st)

View Slide

ž Opening in matlab is done using the
following command:
ž >>bw2=imopen(bw,st)

View Slide

ž Closing in matlab is done using the
following command:
ž >>bw2=imclose(bw,st)

View Slide


View Slide

ž cvErode(src, dst)
ž cvDilate(src, dst)
ž Opening & closing: use the appropriate
sequence

View Slide

ž By default, OpenCV uses the zero
structuring element (all are zeros)
ž You can explicitly specify your
structuring element as well
ž Check the OpenCV Documentation for
more information

View Slide


View Slide

ž Computers can manipulate images
very efficiently
ž But, comprehending an image with
millions of colors is tough
ž Solution: Figure out interesting
regions, and process them

View Slide


View Slide

ž Each pixel is checked for its value
ž If it lies within a range, it is marked as
“interesting” (or made white)
ž Otherwise, it’s made black
ž Figuring out the range depends on
lighting, color, texture, etc

View Slide

ž Demo thresholdRGB
ž Demo thresholdHSV

View Slide


View Slide

ž MATLAB provides a facility to execute
multiple command statements with a
single command. This is done by
writing a .m file
ž Goto File > New > M-file
ž For example, the graythresh function
can be manually written as a m-file as:

View Slide


View Slide

ž Observe that, comments (in green) can
be written after the symbol ‘%’. A
commented statement is not considered
for execution
ž M-files become a very handy utility for
writing lengthy programs and can be
saved and edited, as and when required
ž We shall now see, how to define your own
functions in MATLAB.

View Slide

ž Functions help in writing organized
code with minimum repetition of logic
ž Instead of rewriting the instruction set
every time, you can define a function
ž Syntax:
ž Create an m-file and the top most
statement of the file should be the
function header
ž function [return values] = function-
name(arguments)

View Slide

ž The inbuilt graythresh function in
matlab is used for thresholding of
grayscale images
ž It uses the Otsu’s Method Of
thresholding
ž A sample thresholding opreation has been
shown in the next slide

View Slide

Image thresholded for
the colour blue

View Slide

The Real thing J

View Slide

ž Thresholding of a grayscale image can
be done in MATLAB using the following
commands:
ž >> level=graythresh(imGRAY);
ž >> imBW = im2bw(imGRAY,level);
ž >> imview(imBW);

View Slide

ž The graythresh command basically gives
an idea as to what exactly the threshold
value should be
ž Graythresh returns a value that lies in the
range 0-1
ž This gives the level of threshold which is
obtained by a complex method called the
Otsu’s Method of Thresholding

View Slide

ž This level can be converted into pixel
value by multiplying by 255
ž Lets say, level=.4
ž Then threshold value for the grayscale
image is:
ž 0.4 x 255 =102

View Slide

ž What this indicates is that for the given
image the values below 102 have to
be converted to 0 and values from
103-255 to the value 1
ž Conversion from grayscale to binary
image is done using the function:
ž >>imBW = im2bw(imGRAY,level);

View Slide

ž Here level is the threshold level
obtained from graythresh function
ž This function converts pixel intensities
between 0 to level to zero intensity
(black) and between level+1 to 255 to
maximum (white)

View Slide


View Slide

ž In order to threshold an RGB colour
image using the graythresh function, the
following have to be done:
› Conversion of the RGB image into its 3
grayscale components
› Subtracting each of these components from
the other 2 to get the pure colour intensities
› Finding level for each of the grayscale using
graythresh
› Thresholding the image using imbw and the
level

View Slide

ž Commands:
ž Im=Imread(‘rgb.jpg’);
ž R = im(:,:,1); --Red
ž G = im(:,:,2); --Green
ž B = im(:,:,3); --Blue

View Slide

ž Ronly=R-B-G; --Pure RED
ž Gonly=G-R-B; --Pure GREEN
ž Bonly=B-G-R; --Pure BLUE

View Slide

ž Levelr=graythresh(Ronly);
ž Levelg=graythresh(Gonly);
ž Levelb=graythresh(Bonly);

View Slide

ž Rthresh=im2bw (Ronly,levelR);
ž Gthresh=im2bw(Gonly,levelG);
ž Bthresh=im2bw(Bonly,levelB);

View Slide

ž Using a manually designed thresh_tool
function to adjust the levels as
required
ž To get a feel of how levels vary

View Slide

s=size(im);
temp=im;
thresh=128;
for i=1:s(1,1)
for j=1:s(1,2)
if temp(i,j)temp(i,j)=0;
else
temp(i,j)=255;
end
end
end
imview(temp);

View Slide

ž Splitting of HSV image into
components
ž Using the Hue channel and
thresholding it for different values
ž Since the hue value of a single colour
is constant it is relatively simple to
threshold and gives better accuracy

View Slide

function [temp] = ht(im,level1,level2)
s=size(im);
temp=im;
for i=1:s(1,1)
for j=1:s(1,2)
if (temp(i,j)level1)
temp(i,j)=1;
else
temp(i,j)=0;
end
end
end
imview(temp);

View Slide

ž To this function we give the input
arguments as the upper and lower
bounds of the threshold levels
ž These levels can be obtained by
having a look at the range of hue
values for the particular colour

View Slide

Now that you know the
basics

View Slide

ž cvThreshold(src, dst, threshold, max,
type)
ž type:
› CV_THRESH_BINARY
› CV_THRESH_BINARY_INV
› And several others (check documentation)

View Slide

ž cvInRangeS(src, scalarLower,
scalarUpper, dst);
ž scalarLower = cvScalar(chan1, chan2,
chan3, chan4);
ž scalarUpper = cvScalar(chan1, chan2,
chan3, chan4);

View Slide

ž Ultra basics: motors, drives, etc
ž Digital image representation
ž Color spaces
ž Inter-conversion of color spaces
ž Electronics
ž Filtering
ž Thresholding
ž Morphology

View Slide


View Slide

ž After thresholding, we get a binary
image
ž We want useable information like
centers, outlines, etc
ž There geometrical properties can be
found using many methods. We’ll talk
about moments and contours only.

View Slide

ž Moments are a mathematical concept
ž ∑ ∑intensity*xxorder*yyorder

View Slide

ž Consider xorder=0 and yorder=0 for a
binary image
ž So you’re just summing up pixel values
ž This means, you’re calculating the area
of the white pixels

View Slide

ž Now consider xorder=1 and yorder=0
for a binary image
ž You sum only those x which are white
ž So you’re calculating the numerator of
an average

View Slide

ž The number of points where the pixel
is white is the area of the image
ž So, dividing this particular moment
(xorder=1, yorder=0) by the earlier
example (xorder=0, yorder=0) gives
the average x
ž This is the x coordinate of the centroid
of the blob

View Slide

ž Similarly, for xorder=0 and yorder=1,
you’ll get the y coordinate

View Slide

ž The order of a moment =
xorder+yorder
ž So, the area is a zero order moment
ž The centroid coordinate = a first order
moment / the zero order moment

View Slide

ž There are entire books written on this
topic
ž You can find complex geometrical
properties, like the eccentricity of an
ellipse, radius of curvature of objects,
etc
ž Also check for Hu invariants if you’re
interested

View Slide

Centroid Area etc

View Slide

ž These are pixels of an image that are
conencted to each other forming
separate blobs in an image
ž They can be seperated out and labelled

View Slide

ž >>L = bwlabel(BW,n)
ž Returns a matrix L, of the same size as
BW, containing labels for the connected
objects in BW
ž n can have a value of either 4 or 8, where
4 specifies 4-connected objects and 8
specifies 8-connected objects; if the
argument is omitted, it defaults to 8

View Slide

ž STATS = regionprops(L,properties)
ž Measures a set of properties for each
labeled region in the label matrix L
ž The set of elements of L equal to 1
corresponds to region 1; the set of
elements of L equal to 2 corresponds
to region 2; and so on

View Slide

ž 'Area'– The actual number of pixels in the
region
ž 'Centroid'-- The center of mass of the region.
Note that the first element of Centroid is the
horizontal coordinate (or x-coordinate) of the
center of mass, and the second element is
the vertical coordinate (or y-coordinate)
ž 'Orientation' -- Scalar; the angle (in degrees)
between the x-axis and the major axis of the
ellipse that has the same second-moments
as the region. This property is supported
only for 2-D input label matrices

View Slide

ž BW = imread('text.png');
ž L = bwlabel(BW);
ž stats = regionprops(L,'all');

View Slide

ž Label into an RGB image for better
vizualization
ž RGB = label2rgb(L)

View Slide

ž Binary area open remove small objects
ž BW2 = bwareaopen(BW,P)
ž Removes from a binary image all
connected components (objects) that
have fewer than P pixels, producing
another binary image, BW2.

View Slide


View Slide

ž OpenCV supports functions to calculate
moments upto order 3
CvMoments *moments = (CvMoments*)malloc(sizeof
(CvMoments));
cvMoments(img, moments, 1);

View Slide

ž cvGetSpatialMoment(moments, xorder,
yorder)
ž cvGetCentralMoment(moments, xorder,
yorder)
ž Central = spatial/area

View Slide

ž Example

View Slide

ž For robotics purposes, moments are
fine till have one single object
ž If we have multiple objects in the same
binary image

View Slide


View Slide

ž You can think of contours as an
approximation of a binary image

View Slide

ž You get polygonal approximation of
each connected area

View Slide

ž The output you get for the previous
binary image is:
› Four “chains” of points
› Each chain can have any number of points
› In our case, each chain has four points

View Slide

Contour plotting

View Slide

ž Contour plot of an image im can be made
in MATLAB using the command:
ž im = imread(‘img.jpg');
ž imcontour(im,level)
ž Level=number of equally spaced contour
levels
ž if level is not given it will choose
automatically

View Slide

ž OpenCV linked lists to store the
“chains”
ž We’ll see some code to find out the
squares in the thresholded image you
saw

View Slide

CvSeq* contours;
CvSeq* result;
CvMemStorage *storage = cvCreateMemStorage(0);
• The chains are stored in contours
• result is a temporary variable
• storage is for temporary memory allocation

View Slide

cvFindContours(img, storage, &contours, sizeof(CvContour),
CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));
• img is a grayscale thresholded image
• storage is for temporary storage
• All chains found would be stored in the contours sequence
• The rest of the parameters are usually kept at these values
• Check the OpenCV documentation for details information about the
last four variables

View Slide

while(contours)
{
result = cvApproxPoly(contours, sizeof(CvContour),
storage, CV_POLY_APPROX_DP,
cvContourPerimeter(contours)*0.02, 0);
• The previous command makes contours point to the first chain
• We’re approximating the contour right now
• After this command, result stores the approximate contour as a
polygon (many points)

View Slide

if(result->total==4)
{
CvPoint *pt[4];
pt[i] = (CvPoint*)cvGetSeqElem(result, i);
}
• We’re looking for quadrilaterals, so we check if the number of
points in this particular polygon is 4
• Then, get extract each point using the command cvGetSeqElem
• Once you have the points, you can actually check the shape of the
object as well (by checking angles, lengths, etc)

View Slide

// Do whatever you want with the 4 points
contours = contours->h_next;
}
• Do whatever you want to do with the four points
• Then, we move onto processing the next contour

View Slide


View Slide

ž MATLAB has an image acquisition
toolbox which helps capture images
ž Now-a-days most of the cameras are
available with USB interface
ž Once you install the driver for the
camera, the computer detects the
device whenever you connect it

View Slide

ž In MATLAB, you can check if the
support is available for your camera
ž MATLAB has built-in adaptors for
accessing these devices
ž An adaptor is a software that MATLAB
uses to communicate with an image
acquisition device

View Slide

ž COMMANDS:
ž >> imaqhwinfo
ž >> cam=imaqhwinfo;
ž >> cam.InstalledAdaptors

View Slide


View Slide

ž To get more information about the
device, type
ž >>dev_info =
imaqhwinfo('winvideo',1)
ž Instead of ‘winvideo’, if imaqhwinfo
shows another adaptor, then type that
adaptor name instead of ‘winvideo’.

View Slide


View Slide

ž You can preview the video captured by
the image by defining an object and
associate it with the device
ž >>vid=videoinput(‘winvideo’,1,‘RGB24
_320x240’)

View Slide


View Slide

ž Now to see the video insert the
following command:
ž >> preview(vid)

View Slide

ž You should see a window pop-up, that
displays what your camera is capturing

View Slide

ž The camera may support multiple video
formats. To see for yourself all the
supported formats, type
ž >>dev_info =
imaqhwinfo('winvideo',1);
ž >>celldisp(dev_info.SupportedForma
ts);
ž Check out for yourself the display of other
formats, by replacing `RGB24_320x240`
with other formats, in the definition of the
object vid

View Slide

ž Now to capture an image from the
video, define the object vid as
described before and use getdata to
capture a frame from the video
ž >>start(vid); % This command
initiates capturing of frames and stores
the frames in memory
ž >>im=getdata(vid,1);
ž >>imview(im);

View Slide

ž You can store the captured image as a
.jpg or .gif file using imwrite function
ž >>imwrite(im,'testimage.gif');
ž The image will be stored in
‘MATLAB71\work’ folder

View Slide


View Slide

ž Static Processing
ž Step Processing
ž Real-Time Processing

View Slide

ž Take a single picture of the arena and
process it
ž Find out critical regions and points
ž Apply some geometry and
mathematical calculations
ž Then blindly follow a specified path

View Slide

ž Advantages
› Simplest to implement
› Can be fairly accurate with stepper motors
ž Disadvantages
› Bot goes blind because only one pic
determines the bot motion
› Accuracy is very low especially with DC
motors

View Slide

ž Take images of the arena in discrete
intervals (say Eg. 10secs/image)
ž Process the images and find out critical
regions and points
ž Check bot orientation at these intervals
and try to correct
ž Partial feedback mechanism is
implemented

View Slide

ž Advantages
› Quite simple to implement
› Can be very accurate with stepper motors
ž Disadvantages
› Bot goes blind for a particular period of
time
› Accuracy is compromised with DC motors
› Dynamic environmental changes cannot be
accounted for

View Slide

ž Take images of the arena continuously at
a particular frame rate
ž Process the images and find out critical
regions and points
ž Check bot orientation at every frame and
correct
ž Complete feedback mechanism is
implemented

View Slide

ž Advantages
› Very accurate for both DC and Stepper motors
› Gives dynamic feedback and accounts for
changing environment
› Can give bot orientation at each point of time
ž Disadvantages
› Requires more processing power
› Requires more memory for taking so many
images

View Slide

ž Real-time image processing is the best
approach for any real application
ž Dynamic feedback systems give
excellent accuracy and precision and
hence is the best approach

View Slide

ž Image processing is an important tool in
many applications
ž Problem is that one needs to acquire
images and pre-process them before
doing actual IP
ž Sometimes it may be required that offline
image processing is not possible i.e. one
needs to proceed with real-time IP or
even more, processing of the video itself

View Slide


View Slide

ž Every time you want to capture an
instantaneous image, you have to stop
the video, start it again and use the
getdata function
ž To avoid this repetitive actions, the
Image Acquisition toolbox provides an
option for triggering the video object
when required and capture an
instantaneous frame

View Slide

ž Create an m-file with following
sequence of commands:

View Slide

vid=videoinput('winvideo',1);
triggerconfig(vid,'manual');
set(vid,'FramesPerTrigger',1 );
set(vid,'TriggerRepeat', Inf);
start(vid);

View Slide

for i=1:5
trigger(vid);
im= getdata(vid,1);
figure,imshow(im);
end
stop(vid);
delete(vid);
clear vid;

View Slide

ž In the above code, object im gets
overwritten while execution of each of
the interations of the for loop
ž To be able to see all the five images,
replace im with im(:,:,:,i)

View Slide

ž triggerconfig sets the object to manual
triggering, since its default triggering is
of type immediate
ž In immediate triggering, the video is
captured as soon as you start the object
‘vid’
ž The captured frames are stored in
memory. Getdata function can be used to
access these frames
ž But in manual triggering, you get the
image only when you ‘trigger’ the video

View Slide

ž ‘FramesPerTrigger’ decides the number
of frames you want to capture each time
‘trigger’ is executed
ž TriggerRepeat has to be either equal to
the number of frames you want to
process in your program or it can be set
to Inf
ž If set to any positive integer, you will
have to ‘start’ the video capture again
after trigger is used for those many
number of times

View Slide

ž Once you are done with acquiring of
frames and have stored the images,
you can stop the video capture and
clear the stored frames from the
memory buffer, using following
commands:
ž >>stop(vid);
ž >>delete(vid);
ž >>clear vid;

View Slide

ž Getsnapshot function returns one
image frame and is independent of
FramesPerTrigger property
ž So if you want to process your images
in real-time, this is all you need:

View Slide

vid=videoinput(‘winvideo’,1)
triggerconfig(vid,'manual');
set(vid,'FramesPerTrigger',1);
set(vid,'TriggerRepeat', Inf);
start(vid);
while(1)
{
trigger(vid);
im= getdata(vid,1);
% write your image processing algorithm here
%
% you may break this infinite while loop if a certain
condition is met
}

View Slide


View Slide

Using MATLAB

View Slide

ž MATLAB provides support to access
serial port (also called as COM port)
and parallel port (also called as printer
port or LPT port) of a PC
ž MATLAB has an adaptor to access the
parallel port (similar to adaptor for
image acquisition)

View Slide

ž To access the parallel port in MATLAB,
define an object
ž >> parport=
digitalio('parallel','LPT1');
ž You may obtain the port address using,
ž >> get(parport,'PortAddress')
ž >> daqhwinfo('parallel'); % To get
data acquisition hardware
information

View Slide

ž You have to define the pins 2-9 as output
pins, by using addline function
ž >> addline(parport, 0:7, 'out')
ž Now put the data which you want to
output to the parallel port into a matrix;
e.g.
ž >> dataout = logical([1 0 1 0 1 0 1
1]);

View Slide

ž Now to output these values, use the
putvalue function
ž >> putvalue(parport,dataout);
ž Alternatively, you can write the decimal
equivalent of the binary data and output
it
ž >> data = 23;
ž >> putvalue(parport,data);

View Slide

ž You can connect the pins of the parallel
port to the driver IC for the left and
right motors of your robot, and control
the left, right, forward and backward
motion of the vehicle
ž You will need a H-bridge for driving the
motor in both clockwise and anti-
clockwise directions

View Slide

Using MATLAB

View Slide

ž Things are a little more involved
ž There is this library called inpout32
ž Makes your task really simple
ž Just follow the instructions that come
along, and you’ll be sending data to
your robot!

View Slide

ž Or you could use the following code
ž The idea is:
› Create a virtual file that “represents” the
port itself (parallel, serial, etc)
› Keep this file open
› And keep writing to this file
› So, data is automatically send to the
desired port

View Slide

ž Step 1: Create a global variable named
hPort
HANDLE hPort;

View Slide

ž Step 2: Create function to create the
virtual file, and store its “handle” in hPort
(contd)
bool SerialOpen(LPCWSTR strPort)
{
// Open the serial port.
hPort = (HANDLE)CreateFile (strPort, // Pointer to the name of the port
GENERIC_READ | GENERIC_WRITE, // Access (read-write) mode
0, // Share mode
NULL, // Pointer to the security attribute
OPEN_EXISTING, // How to open the serial port
0, // Port attributes
(long)NULL); // Handle to port with attribute
// to copy
DCB PortDCB;
DWORD dwError;
// Initialize the DCBlength member.
PortDCB.DCBlength = sizeof (DCB);
// Get the default port setting information.
GetCommState (hPort, &PortDCB);

View Slide

ž Step 2: Some more port configuration
(contd)
// Change the DCB structure settings.
PortDCB.BaudRate = 9600; // Current baud
PortDCB.fBinary = TRUE; // Binary mode; no EOF check
PortDCB.fParity = TRUE; // Enable parity checking
PortDCB.fOutxCtsFlow = FALSE; // No CTS output flow control
PortDCB.fOutxDsrFlow = FALSE; // No DSR output flow control
PortDCB.fDtrControl = DTR_CONTROL_ENABLE; // DTR flow control type
PortDCB.fDsrSensitivity = FALSE; // DSR sensitivity
PortDCB.fTXContinueOnXoff = TRUE; // XOFF continues Tx
PortDCB.fOutX = FALSE; // No XON/XOFF out flow control
PortDCB.fInX = FALSE; // No XON/XOFF in flow control
PortDCB.fErrorChar = FALSE; // Disable error replacement
PortDCB.fNull = FALSE; // Disable null stripping
PortDCB.fRtsControl = RTS_CONTROL_ENABLE; // RTS flow control
PortDCB.fAbortOnError = FALSE; // Do not abort reads/writes on error
PortDCB.ByteSize = 8; // Number of bits/byte, 4-8
PortDCB.Parity = NOPARITY; // 0-4=no,odd,even,mark,space
PortDCB.StopBits = ONESTOPBIT; // 0,1,2 = 1, 1.5, 2

View Slide

ž Step 2: And finally…
// Configure the port according to the specifications of the DCB
// structure.
if (!SetCommState (hPort, &PortDCB))
{
// Could not configure the serial port.
dwError = GetLastError();
printf("Serial port creation error: %d", dwError);
MessageBox(NULL, L"Unable to configure the serial port", L"Error", MB_OK);
return false;
}
return true;
}

View Slide

ž This function works equally well for
both serial ports and parallel ports

View Slide

ž Step 2: I’ve assumed you’re creating a
function
ž This function returns a true when the
port is created successfully
ž If you’re not, replace the “return”
statements with “printf”s

View Slide

ž Example usage of this function
› hPort = SerialOpen(L”COM8:”); // Serial Port
› hPort = SerialOpen(L”LPT1:”); // Parallel
ž This is actually how you can access ports
in DOS as well… using COM8: and LPT1:
instead of C:, D:, etc
ž The L before the quotes is just syntax for
C/C++

View Slide

ž Step 3: Next, we’ll write functions to
write to the virtual file we’ve created
bool SerialWrite(byte theByte)
{
// The port wasn't opened
if(!hPort)
return false;
DWORD dwError, dwNumBytesWritten;
WriteFile (hPort, // Port handle
theByte, // Pointer to the data to write
1, // Number of bytes to write
&dwNumBytesWritten, // Pointer to the number of bytes written
NULL // Must be NULL
);
return true;
}

View Slide

ž You pass the byte you want to write as
a parameter, and it gets written to the
port
ž Example: SerialWrite(12)

View Slide

ž Step 4: And finally, a function to close
the port
ž Example: SerialClose()
bool SerialClose(void)
{
if(!hPort)
return false;
CloseHandle(hPort);
return true;
}

View Slide

ž Again, I’ll emphasize that these
functions will work equally well for
both parallel ports and serial ports

View Slide


View Slide

ž A critical part of image segmentation
ž Used for detecting meaningful
discontinuities in intensity values
ž Done by finding first and second order
derivatives of the image
ž Also known as gradient of the image

View Slide

ž There are different kinds of edge detectors
based on the criteria of derivatives
ž An edge detector can be sensitive to
horizontal or vertical lines or both
ž In detection we try to find out regions
where the derivative or gradient is greater
than a specified threshold

View Slide

ž General Command in MATLAB
ž [g t] = edge(im,’method’,parameters);
ž g-gradient, t-threshold value

View Slide

ž Different methods of edge detection

View Slide

ž A threshold value can be given
manually as argument
ž g=edge(im,’method’,t);
ž t-threshold
ž Sobel& Canny are the more frequently
used edge detectors

View Slide


View Slide

ž Image Acquisition
ž Image Cropping
ž Image Filtering
ž Image Segmentation
ž Image Thresholding
ž Finding critical points (Centroids)
ž Finding bot centroidand orientation
ž Robot Feedback control
ž Robot Control

View Slide

ž Static processing
ž Step processing
ž Real-time processing

View Slide

ž Manual Cropping:
› I2 = imcrop(I);
ž Coordinate Cropping:
› I2 = imcrop(I,[x1 y1 x2 y2]);

View Slide

ž Mean filter
ž Median filter
ž Histogram equalization

View Slide

ž Use different kinds of edge detections
› Sobel
› Canny

View Slide

ž Thresholding
› In RGB
› In HSV
ž Separating colors

View Slide

ž Finding areas of blobs
ž Finding centroids of blobs

View Slide

ž Using orientation

View Slide

ž Control using parallel port

View Slide


View Slide

ž The idea of an orientation tag is to
precisely indicate the orientation of the
robot
ž It should happen with the least
number of operations

View Slide

ž This orientation tag is bad
ž You can tell the position, but not the
direction the robot is facing

View Slide

ž This orientation tag is excellent for a
single bot

View Slide

ž But if you have a team of bots, this
won’t be the best choice
ž You need to have multiple colors for
each bot
ž So, you have more operations
ž Slowing down the program

View Slide

ž A better choice for a team of bots
ž The asymmetry helps distinguish
between multiple bots, with just two
colors

View Slide

ž Some more orientation tags

View Slide


View Slide

ž With C/C++ you get two methods to
capture images:
› Using OpenCV’s built in libraries
› Using some 3rd party capturing library

View Slide

ž Use OpenCV functions
ž Use DirectX functions (on Windows)

View Slide

Images, served in C/C++

View Slide

ž OpenCV lets you access cameras
through the CvCapture structure
ž So we create a CvCapture structure

View Slide


View Slide

ž Tries to get access to cam #index
ž Useful when you have multiple
cameras attached to the same machine
(like in stereo vision)
ž Then we check if we were able to get
exclusive control of the camera. If not,
quit.

View Slide

ž This window is for our convenience
ž Displaying what’s going on within the
program, what decisions are taken, etc

View Slide


View Slide

ž We’ll create a function to take snapshots
ž It will use the CvCapture structure to tap
into the camera’s stream
ž It will return a single frame as an
IplImage structure
ž Scroll down, and add the function

View Slide


View Slide

ž Now, we’ll display a live stream in the
window we had created
ž Because the getSnapshot() function
returns a single frame, we need to take
snaps regularly
ž So it goes into the do…while loop

View Slide


View Slide

ž If you try compiling the program right
now, you’ll get an error
ž getSnapshot() comes “after” the main
function
ž So the compiler doesn’t know if it
exists
ž Hence we need the so called
“prototype”

View Slide


View Slide

ž Once we’re done, we need to release

View Slide


View Slide

ž The cvCaptureFromCam works only for
some supported cameras
ž DirectX is a much bigger library and
supports almost all cameras that exist

View Slide

ž First, download the Microsoft Platform
SDK for Windows Server 2003 R2
› http://www.microsoft.com/downloads/deta
ils.aspx?familyid=E15438AC-60BE-41BD-
AA14-7F1E0F19CA0D&displaylang=en

View Slide

ž Second, download the DirectX SDK
ils.aspx?FamilyID=77960733-06e9-47ba-
914a-844575031b81&DisplayLang=en

View Slide

ž Third, download the DirectShow SDK
ils.aspx?FamilyID=8af0afa9-1383-44b4-
bc8b-7d6315212323&DisplayLang=en

View Slide

ž Fourth, download the VideoInput
library
› http://muonics.net/school/spring05/videoI
nput/

View Slide

ž These are external libraries. So we need
to tell Visual Studio where to find the
required external files
ž So we follow steps similar to the OpenCV
ones.
ž The VideoInput package comes with lots
of sample code, so you shouldn’t have
much problem

View Slide

ž You can check some sample code at
this website as well
› http://opencv.willowgarage.com/wiki/Direc
tShow

View Slide


View Slide

ž For real time, you need to process
captured frames as quickly as possible
ž So we use a loop of some kind (usually
a do…while)
ž Within the loop, you do the following
tasks:

View Slide

ž Task 1: Capture an image
ž Task 2: Pre–processing it
ž Task 3: Process the image
ž Task 4: Take a decision
ž Task 5: Move the bot (with feedback)
ž Task 6: Go to Task 1

View Slide

ž Quite obvious
ž You need an image
ž Only then can you process something

View Slide

ž Once you have the image, you need to
enhance it with pre-processing
ž Increase contrast, reduce noise,
smoothen it out, etc

View Slide

ž With the pre-processed image, you
figure out the location of each object in
the arena
ž Use moments, contours, or anything
else
ž Thresholding, morphology, etc are
helpful here

View Slide

ž With the location of each object, you
can decide what to do next
ž Bot: Should I go to the red ball
because it’s the closest? Or should I go
to the green ball because it has the
maximum number of points

View Slide

ž Once decided where to go, you make
the robot move
ž You must include feedback in this step
itself (maybe another do…while loop)

View Slide

ž You have two options:
› Check if further movement is necessary
(have all the balls been potted?) If
required, only then go to Task 1
› Blindly go to Task 1 (the decision module
will tell the bot when to stop)
ž Both options are good enough

View Slide

ž Task 1: Capturing the image
ž Task 2: Pre-processing the image
ž Task 3: Processing the image

View Slide

ž Task 4: Taking a decision
ž Task 6: Go to Task 1

View Slide

ž Task 5: Move the bot (with feedback)
ž We won’t go into code. Just high level
logic on how to go about it

View Slide

ž For the simplest feedback mechanism,
you use pixels
ž Everything is measure in terms of
pixels: distances, coordinates, etc
ž Angles are usually represented in
radians (cos, sin, etc work well with
radians)

View Slide

ž First, you need to decide your
coordinate system
ž And you need to stick with it
throughout your code
ž Here’s a coordinate system I used
several times

View Slide

ž This is a top-down view of the arena,
just like the camera sees it

View Slide

ž So all your calculations MUST use this
particular coordinate system
ž In particular, you must make sure all
your angle calculations are consistent
ž And that they cycle through 359-1
degrees perfectly

View Slide

ž Possible function you might want to
create:
› MoveBotToPosition(x, y)
› TurnBotToAngle(angle)

View Slide

ž TurnBotToAngle(angle)
› Turns the bot to angle degrees of the
coordinate system
› This would be the very basic feedback
function
› Within this function, you have a loop
› This loop keeps running as long as the bot
isn’t oriented at angle degrees
› You’ll take snapshots within this function
as well

View Slide

ž MoveBotToPosition(x,y)
› Moves the bot until it reaches the
coordinates (x,y) in the image
› If required, you can also put a call to
TurnBotToAngle (to orient the bot to move)
› Again, there’s a loop which keeps running
until the bot reaches the desired position
› And you’ll need to take multiple snapshots
to check where the bot actually is

View Slide

ž You obviously need to set a range
ž If you say TurnBotToAngle(30), it’s
very unlikely that the bot will orient to
exactly 30 degrees
ž A range, say 28-32 should be good
enough

View Slide

ž Reminder: All x, y and angle we’ve
talked about are in the image, in terms
of pixels
ž We have no idea how they relate to
physical distances and angles
ž But we’re sure that they are
proportional to physical distances and
angles

View Slide

ž Though, you CAN calibrate your
camera and actually figure out physical
distances
ž For example, 5pixels = 3cm
ž This is very much possible, but of no
use for our purposes!

View Slide

ž Use classes and structures as much as
possible. And you don’t need to know
OOPs to use them.
ž They really simplify your work, and
even make the code more readable

View Slide

ž What we’ve described requires that the
bot stops and then checks if the angle
is correct or not, etc
ž Try working on something which
checks the bot’s angle without
stopping the bot

View Slide

ž Working in the YUV space (this is what
cameras use… so thresholding in YUV
itself will eliminate processing time
consumed by YUV to RGB conversion)

View Slide

ž Kalman filters: If a bot is static, still it’s
position might be calculated as
different for different frames.
ž If you use Kalman filters, you can
“smooth out” the position data and get
precise positioning and angle data

View Slide

ž Can you eliminate thresholding
altogether?
ž Yes you can!!!
ž How, you ask? Figure it out! Its
EXTREMELY simple!

View Slide


View Slide

ž You now know enough to participate in
image processing based competitions
ž All this knowledge can even serve as a
start point for further studies in image
processing
ž Enjoy!

View Slide

ž Utkarsh Sinha
ž Ajusal Sugathan
ž Oh, btw, visit http://liquidmetal.in/ for
more!

View Slide


View Slide

RoboCV Workshop

RoboCV Workshop

Utkarsh Sinha

Other Decks in Technology

Featured

Transcript