This part focuses on Video Analysis.
1. Meanshift
1.1 Concept
Consider the example:
The blue circle "C1", its original center is blue rectangle "C1_o". Find the centroid of the points inside that window "C1_r" (small blue circle). Circle center and centroid do not match. Move blue circle such that center of new circle matches with previous centroid. Again find the new centroid. repeat the process until center of new circle and its centroid falls on the same location. It is green circle "C2" and it has maximum number of points (maximum pixel distribution). This is meanshift algorithm.
1.2 Meanshift in OpenCV
OpenCv provides cv2.meanShift() and we have to do:
+ Setup the target
+ Provide initial location of window
+ Find its histogram (only Hue is considered) so that we can backproject the target on each frame for meanshift calculation.
2. Camshift
2.1 Concept
There is a problem with previous demo. Our window always has the same size when object is farther away and it is very close to camera. We need to adapt the window size with size and rotation of the target. We use camshift. It updates the size of the window and also calculates the orientation of best fitting ellipse to it. Then it applies the meanshift with new scaled search window.
2.2 Camshift in OpenCV
OpenCV provides cv2.CamShift() returns a rotated rectangle and box parameters.
Let 's make a demo using Camshift:
Tracking the bird in Flappy Bird game. Here are resources
+ The video for the demo is here.
+ Flappy bird template for using SIFT feature matching:
3. Optical Flow
3.1 Concept
Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera. Optical flow has assumptions:
+ The pixel intensities of an object do not change between consecutive frames.
+ Neighbor pixels have similar motion.
Optical flow has many applications in areas like :
+ Video Compression
+ Video Stabilization
+ Measure the velocities of objects in the video. In general, moving objects that are closer to the camera will display more apparent motion than distant objects that are moving at the same speed.
$f_{x}u+f_{y}v+f_{t}=0\\
f_{x}=\frac{\delta f}{\delta x};f_{y}=\frac{\delta f}{\delta y}\\
u=\frac{\delta x}{\delta t};v=\frac{\delta y}{\delta t}$
$f_{x};f_{y}$ are image gradient, $f_{y}$ is time gradient. $(u,v)$ are unknown.
There are several methods to solve this problem.
3.2 Lucas-Kanade method
It assumes that the flow is essentially constant in a local neighborhood of the pixel under consideration, and solves the basic optical flow equations for all the pixels in that neighborhood, by the least squares criterion.
3.3 Optical Flow in OpenCV
OpenCV provides cv2.calcOpticalFlowPyrLK(). We have to do:
+ Using cv2.goodFeaturesToTrack() to find points to track.
+ Pass the previous frame, previous points and next frame to cv2.calcOpticalFlowPyrLK()
+ Function returns next points and status number with value of 1 if next point is found, else zero
Apply this for the Flappy Bird video above:
4. Background Subtraction
Consider the application which counts visitors or vehicles. We need to extract the person or vehicles alone or extract the moving foreground from static background. OpenCV provides some methods:
4.1 BackgroundSubtractorMOG
It is a Gaussian Mixture-based Background/Foreground Segmentation Algorithm. This method uses the characteristic that the probable background colours are the ones which stay longer and more static.
4.2 BackgroundSubtractorMOG2
This method improves the adaptability to varying scenes due illumination changes compare to previous method.
1. Meanshift
1.1 Concept
Consider the example:
1.2 Meanshift in OpenCV
OpenCv provides cv2.meanShift() and we have to do:
+ Setup the target
+ Provide initial location of window
+ Find its histogram (only Hue is considered) so that we can backproject the target on each frame for meanshift calculation.
2. Camshift
2.1 Concept
There is a problem with previous demo. Our window always has the same size when object is farther away and it is very close to camera. We need to adapt the window size with size and rotation of the target. We use camshift. It updates the size of the window and also calculates the orientation of best fitting ellipse to it. Then it applies the meanshift with new scaled search window.
2.2 Camshift in OpenCV
OpenCV provides cv2.CamShift() returns a rotated rectangle and box parameters.
Let 's make a demo using Camshift:
Tracking the bird in Flappy Bird game. Here are resources
+ The video for the demo is here.
+ Flappy bird template for using SIFT feature matching:
Figure: Flappy bird template
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | import numpy as np import cv2 MIN_MATCH_COUNT = 10 cap = cv2.VideoCapture('flappybird.mp4') ret,frame = cap.read() img2 = cv2.imread('bird.png') # trainImage gray1= cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY) gray2= cv2.cvtColor(img2,cv2.COLOR_BGR2GRAY) # Initiate SIFT detector sift = cv2.xfeatures2d.SIFT_create() # find the keypoints and descriptors with SIFT kp1, des1 = sift.detectAndCompute(gray1,None) kp2, des2 = sift.detectAndCompute(gray2,None) # BFMatcher with default params bf = cv2.BFMatcher() matches = bf.knnMatch(des1,des2, k=2) # Apply ratio test good = [] for m,n in matches: if m.distance < 0.5*n.distance: good.append([m]) if len(good)>MIN_MATCH_COUNT: #find the object to track src_pts = np.float32([ kp1[m[0].queryIdx].pt for m in good ]).reshape(-1,1,2) c,r,w,h = cv2.boundingRect(src_pts) track_window = (c,r,w,h) # set up the ROI for tracking roi = frame[r:r+h, c:c+w] hsv_roi = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV) #these range was gotten by trial and error using the Pain tool #https://tech.deepumohan.com/2011/04/how-to-get-html-color-code-from-image.html mask = cv2.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((130.,130.,130.))) roi_hist = cv2.calcHist([hsv_roi],[0],mask,[180],[0,180]) cv2.normalize(roi_hist,roi_hist,0,255,cv2.NORM_MINMAX) # Setup the termination criteria, either 10 iteration or move by atleast 1 pt term_crit = ( cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1 ) while(1): if ret == True: hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV) dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1) # apply meanshift to get the new location ret, track_window = cv2.CamShift(dst, track_window, term_crit) # Draw it on image pts = cv2.boxPoints(ret) pts = np.int0(pts) img2 = cv2.polylines(frame,[pts],True, 255,2) cv2.imshow('img2',img2) k = cv2.waitKey(60) & 0xff if k == 27: break else: cv2.imwrite(chr(k)+".jpg",img2) ret ,frame = cap.read() else: break cv2.destroyAllWindows() cap.release() else: print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT) |
3.1 Concept
Optical flow is the pattern of apparent motion of image objects between two consecutive frames caused by the movement of object or camera. Optical flow has assumptions:
+ The pixel intensities of an object do not change between consecutive frames.
+ Neighbor pixels have similar motion.
Optical flow has many applications in areas like :
+ Video Compression
+ Video Stabilization
+ Measure the velocities of objects in the video. In general, moving objects that are closer to the camera will display more apparent motion than distant objects that are moving at the same speed.
Figure: a ball moving in 5 consecutive frames
The equation of Optical Flow is:$f_{x}u+f_{y}v+f_{t}=0\\
f_{x}=\frac{\delta f}{\delta x};f_{y}=\frac{\delta f}{\delta y}\\
u=\frac{\delta x}{\delta t};v=\frac{\delta y}{\delta t}$
$f_{x};f_{y}$ are image gradient, $f_{y}$ is time gradient. $(u,v)$ are unknown.
There are several methods to solve this problem.
3.2 Lucas-Kanade method
It assumes that the flow is essentially constant in a local neighborhood of the pixel under consideration, and solves the basic optical flow equations for all the pixels in that neighborhood, by the least squares criterion.
3.3 Optical Flow in OpenCV
OpenCV provides cv2.calcOpticalFlowPyrLK(). We have to do:
+ Using cv2.goodFeaturesToTrack() to find points to track.
+ Pass the previous frame, previous points and next frame to cv2.calcOpticalFlowPyrLK()
+ Function returns next points and status number with value of 1 if next point is found, else zero
Apply this for the Flappy Bird video above:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | import numpy as np import cv2 cap = cv2.VideoCapture('flappybird.mp4') # params for ShiTomasi corner detection feature_params = dict( maxCorners = 100, qualityLevel = 0.3, minDistance = 7, blockSize = 7 ) # Parameters for lucas kanade optical flow lk_params = dict( winSize = (15,15), maxLevel = 2, criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03)) # Create some random colors color = np.random.randint(0,255,(100,3)) # Take first frame and find corners in it ret, old_frame = cap.read() old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY) p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params) # Create a mask image for drawing purposes mask = np.zeros_like(old_frame) refind = 0 while(1): ret,frame = cap.read() frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) refind = refind + 1 #re-find feature to track after 100 frames in case object disappears if(refind == 100): old_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params) mask = np.zeros_like(frame) refind = 0 # calculate optical flow p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params) # Select good points good_new = p1[st==1] good_old = p0[st==1] # draw the tracks for i,(new,old) in enumerate(zip(good_new,good_old)): a,b = new.ravel() c,d = old.ravel() mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2) frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1) img = cv2.add(frame,mask) cv2.imshow('frame',img) k = cv2.waitKey(30) & 0xff if k == 27: break # Now update the previous frame and previous points old_gray = frame_gray.copy() p0 = good_new.reshape(-1,1,2) cv2.destroyAllWindows() cap.release() |
Consider the application which counts visitors or vehicles. We need to extract the person or vehicles alone or extract the moving foreground from static background. OpenCV provides some methods:
4.1 BackgroundSubtractorMOG
It is a Gaussian Mixture-based Background/Foreground Segmentation Algorithm. This method uses the characteristic that the probable background colours are the ones which stay longer and more static.
1 2 3 4 5 6 7 8 9 10 11 12 13 | import numpy as np import cv2 cap = cv2.VideoCapture('flappybird.mp4') fgbg = cv2.bgsegm.createBackgroundSubtractorMOG() while(1): ret, frame = cap.read() fgmask = fgbg.apply(frame) cv2.imshow('frame',fgmask) k = cv2.waitKey(30) & 0xff if k == 27: break cap.release() cv2.destroyAllWindows() |
This method improves the adaptability to varying scenes due illumination changes compare to previous method.
1 2 3 4 5 6 7 8 9 10 11 12 13 | import numpy as np import cv2 cap = cv2.VideoCapture('flappybird.mp4') fgbg = cv2.createBackgroundSubtractorMOG2() while(1): ret, frame = cap.read() fgmask = fgbg.apply(frame) cv2.imshow('frame',fgmask) k = cv2.waitKey(30) & 0xff if k == 27: break cap.release() cv2.destroyAllWindows() |
0 Comments