Using iPiSoft markerless motion capture system

 

Specifically for a 3 camera rig for advanced gesture capture

 

Version 1.0 26/11/2009

 

This document is a mix of what I’ve taken from the iPiSoft support forums and my own experience with the system so far.  It is very much a work in progress and if the information in here turns out to be incorrect, incomplete, or misleading, please let me know and I’ll update.

Equipment needed

2* AA Maglite with candle mode and a proper bulb, not LED

3* Logitech QuickCam Pro 9000

3* USB extension leads

3* stands – I used these:

3 X STUDIO LIGHTING FLASH STANDS FOR ELINCHROM/BOWENS

From the ebay seller “outdoorvalue”.  £40 delivered for three stands that are sturdy, decent quality, and easily height adjustable.  Perfect.

 

 

Set the cameras up

Physically

You need to position the three cameras in front/left/right configuration.  You will need a room large enough to position the cameras back far enough so that the whole of the actor can be seen, and even further if the actor is to significantly move around while performing.

 

I mounted the cameras on the stands like so:

Yes, that’s just some masking tape.  Ideal, as it tends to only stick to itself well, everything else it touches lightly.  This is working so far, the only minor drawback is that you need to lift and turn the whole stand to rotate the camera.  Twisting the camera on the tape is a bad idea as it’ll slowly return to the original position during your shoot!

 

Note I have also labelled the camera L for left, and the lead as well, so I easily know which camera is which.  The others are labelled, of course, F for front and R for right, and these directions from the perspective of the actor.

 

If you plug them into the PC/laptop you’re using to record in the order Left, Front, Right, that seems to ensure the recorder software picks them up in the right order.


In software

There is some initial setting up of the webcams before using them.  These settings should be persistent, but it is a good idea to check them before each calibration run, as having any of them revert to defaults could cause your entire session to be junk.

 

This is specifically for the Logitech camera software, but the principles remain the same for whatever camera device you are using.  Each camera needs to be set up individually.  There is an option in the settings that lets you switch between cameras, which we will get to in a moment.

 

 

 

 

 

This is the Quickcam control software.  The only button we are interested in is the cogwheel/gears one, which brings up the options page.

 

 

 

 

 

 

In the Webcam Settings page, make sure the “Auto” checkbox for White Balance is unchecked, and set the white balance all the way to the left.  The actual setting doesn’t matter as we aren’t recording production video, so moving it all the way to one stop makes it easy to be consistent across cameras.

 

It’ll be a good idea to get all the other sliders set to the same value across all your cameras.

 

 

 

 

 

 

 

 

 

 

 

 

 

In the RightLight settings, turn off “Use Automatic RightLight settings”.  You may get warned about ‘needing’ it when you start the recorder software, tell it politely to sod off and not ask again.

 

The gain will be set fully to the right, and that is dealt with by the iPiSoft Recorder software, so don’t worry about that.  Exposure is also controlled from with the Recorder software.  The other settings can be left on their defaults, but again, ensure they are the same across all the cameras you are using.

 

 

 

 

 

 

 

 

 

In the Advanced settings, turn off AntiFlicker.  Obviously we don’t want any mirroring or anything, and the LED control is irrelevant.

 

The camera selector dropdown list is where you choose which camera you are changing the settings for.  Unfortunately, they will all be named the same if the cameras are all the same!  The best thing to do is to simply start with the top one in the list, fully configure it, and then select the next one down and repeat.  Once you’ve done that, go through them all and double check.
Actor clothing

You have to wear clothing that helps the capture software discern not only the actor from the background but different parts of the actor as well – so a tight long sleeve shirt with another t-shirt over the top, in a different colour, helps the software pick out which bits are hands, arms, body, etc.

 

So the recommendations are:

 

And the actual dress should be:

 

Currently I’m using black shoes, blue jeans, black long sleeve shirt, and am switching from a white sleeveless top to a red t-shirt to better fit what the software expects.  I may also try gloves, as the capture got confused when my hand passed in front of my face, perhaps because the colours were the same.

 

One last observation.  You will look an idiot.  Get used to it.

Recording Video

Recording settings

I use the same recording settings for calibration and recording the actual acting footage:

 

Turn the Exposure down as low as you can get away with, as the lower this is the less hard the cameras and software have to work to achieve 30fps

 

Use the “…” button to choose the path where you want to save your file out.  The file size is about 0.5mb a second when recording three cameras at 320*240.

 

Note - I would ideally like to run 640*480 capture but there currently appears to be a technical limitation that I can’t get around.  I can see two cameras at 640*480, or three at 320*240, but as soon as I try three at 640*480 the software locks up.  This has happened on every computer I’ve tried it on, I don’t think it’s a matter of CPU horsepower or anything.

 

Recording Calibration

The calibration step is a vital part of the proceedings.  It tells the Studio software precisely where the cameras are in 3d space.  Without this step, cameras can get out of sync, no matter how closely you think you’ve manually aligned them.  This leads to situations where perhaps two of the three cameras halfway into a gesture are still on track, but the actor is wildly out of frame according to the 3rd camera.  So, it’s a good thing, if a bit of a pain.

 

You will need to perform a calibration video for each recording session unless you are in the fortunate circumstance of having a room you can leave overnight and are 100% sure nothing can move the cameras between sessions.

 

Calibration is performed using the maglite in candle mode (taking the lens off completely).  Try not to have any highly reflective surfaces near that may give false images, and any light sources such as lamps, ceiling lights, laptop/PC screens etc..

 

There are tutorial videos on how to perform calibration, but the basic path I follow is this:

 

The process takes around 30s of actual calibration footage.  This has worked for me on my home PC, and was a disaster on my friend’s PC so perhaps there is still some mystery shrouding calibration.

 

Recording Motion Acting

 

To record your actor, simply start recording, get them to do their thang, and stop recording.  Possibly the easiest part of the whole proceedings!  Just make sure they don’t leave the view of the cameras with any body parts, and get them to do a T-Pose at the start of each capture.

 

It’s probably a good idea to get a folder and file naming convention sorted out.  You’ll need to be able to match up the calibration video file to all the video files of each mocap action taken with that camera setup.  You’ll also need to record the Scene file that’s generated from the calibration, and various capture processing files and the resulting BVH or SMD exports!

Processing the calibration video

Open the iPi Studio software.  Use File – Open to open your calibration video.

The video will start playing.  Pause it using the video control buttons at the bottom of the screen.

Setting up

 

Move the scrubber slider at the bottom of the screen to the point where you are holding the light out front and steady.  Note the frame number, and set the left hand edge of “Region of Interest” to that frame number.

 

Now move the scrubber to the end of the calibration sequence where you are once again holding the light out front and steady.  Note the frame number again, and set the right hand edge of “Region of Interest” to that number.  This tells the software which part of the video it should use as the calibration sequence.

 

Now switch to the Scene tab.  Use the Orbit, Zoom, and Move controls on each camera to roughly position it in the right place.  You click and drag on the buttons to affect the camera.  Switch between camera views using the Camera buttons at the top of the screen.

 

Set the Field of View to 74 degrees if you are using the Logitech Quickcam 9000 cameras.  Other cameras may vary, but this needs to be set correctly!  It is a per-camera setting as well, so make sure all 3 cameras have 74 degrees entered.

 

I turned the ambient light settings right down for the calibration run, as I thought it might reduce stray reflections etc..  It didn’t seem to hurt.

 

Save your scene out using the Save Scene button, with a filename that indicates it’s the basic uncalibrated start point.

 

Calibrating

Switch to the Calibration tab.  Uncheck “Auto detect initial camera positions”.  Uncheck “Audo adjust camera FOV”.

 

Now hit the Calibrate… button.  There are two stages to calibration.  The first pass tracks the light in the video and plots the position of it in 3d space.  Once that’s finished, it then goes through each point and tries to work out the exact position of each camera in relation to the other points.

 

This is a time consuming process.  It can take around 15 to 20 minutes.  During this time, your camera views may go all over the place, including under the ground.  Do not worry too much unless it starts smearing all the points out and turning them all red.  Don’t write off a calibration run until it’s actually finished (a progress bar might be nice in future versions!), although sometimes it’s such a disaster you just know it isn’t going to recover.

 

If you get a successful calibration (the cameras look like they are in about the right place, the positions of the green dots area kinda consistent with what you did, there aren’t too many red dots, that sort of thing), then save out the Scene (Scene tab, Save Scene button) and also save out the mocap file (File menu, Save).  Make a note in the filename that it’s a successful calibration.

 

Note - The Scene file is the camera and actor positions.  The mocap also contains a link to the video, and capture data that’s been plotted or the calibration tracking.

 

Detecting ground

You need to choose at least three ground points to detect where the ground is.  Choose a square or triangular pattern of data points to do this.  You highlight them by scrubbing to the target data point and then clicking on the “Mark as Ground” button in the Calibration tab.  Repeat this for your other target points.

 

You may find that on the 3rd and subsequent selections, the camera angle seems to go askew.  Don’t worry about this too much, it is a visual bug only.  Clicking on a different camera view and then back again will show that the camera is still in the correct place.

 

Once you have three of four data points defined as ground, click on “Detect additional ground points”.  This should then highlight the majority of the remaining ground points.  If something goes wrong at this point, reload the previously saved mocap and scene files.

 

If the ground points are detected correctly, save out the scene and the mocap file again, with yet another unique filename indicating that it’s a successful calibration with ground points.  The Scene file at this point is the one you’ll use when tracking your motion capture videos.
Processing the capture video

 

It’s probably best at this stage to close and reopen the iPi Studio software, as I’ve found that simply opening another project doesn’t necessarily “clear the decks”.

 

Setting up the model

Open your capture video with File – Open off the menu.  Load in your previously saved fully-calibrated and ground-pointed Scene file.  Scrub to a good frame of your actors’ T-Pose.

 

 

Use the Move button to position the model over the actor.  Keep switching the camera views to make sure the model is aligned in all directions.  Once that’s about right, use the IK button and click on the model’s wrists to highlight them.  A yellow cube will appear, which you can click and drag on to get the arms out straight in the T-Pose.

Switch to the Actor tab.  Set up the height of the model, and tweak this so that the model is the same size as the actor in the video.

You can use the crossfading tools to help match the shirt length etc with the video actor’s clothes.

Note - The radio button defaults to “Long Sleeve Shirt” every time this screen is opened, but the previous setting is remebered if you chose the other option.

Once you’ve set all these options, click on “Analyze Actor Appearance”.  It should now recognize what colours the items of clothing are that your actor is wearing and set the model up appropriately.

Use the “Save Actor” button to save out this particular actor.

 

You should now have something that looks a bit like this:

 

Now scrub to your actual action, and exactly like in the calibration sequence, set your Area of Interest to the start and end frames of the acting.  Save your mocap file with yet another filename.

 

Tracking the motion

 

 Switch to the Tracking tab.  This is where the real black magic begins.  Using the Move, IK, and Rotate tools, align the model with the first frame of the actor, much like you did for the T-Pose.  It’s probably a good idea to save the mocap file with yet another filename, to give you a point to go back to if the next step fails.

 

Click on “Refit Pose”.  This should align the model to your actor’s pose more accurately.  If the legs go mad, make sure that the model is actually stood on the ground plane and not slightly under it.  Use the right mouse button in the preview window to rotate the view and check.  Move the model up to the ground plane if necessary (you may need to reload the mocap file to reset the legs).

 

Note - I found that foot tracking caused a lot of jitter problems so until I work out what I’m doing wrong there, I’m leaving it unchecked.

 

Click on “Track Forward” to begin the motion capture process (at last!).  It takes about 5 to 6 seconds a frame on my quad core GTX260 PC.  You are now in the lap of the gods!  Good luck!

 

Smoothing

Should you get a decent capture, after of course saving the mocap file as a “final” version, you can now alter the “Trajectoy Filtering” value.

In the Tracking tab you will find two Trajectory Filtering buttons – Sharpen and Smooth. 

 

These are instant settings, and are applied to the entire region of interest you have set.  It doesn’t need to be “applied” to a section or be set during rendering or anything.  You can see the results immediately by viewing the capture results on a setting of 1, then smoothing it to say 10 and viewing the results again.  A setting of 10 makes a significant difference in the smoothness of the animation.

 

 

Once you have your ideal smoothing in place, all that remains is to export the file to the format of your choice.  My target is the Source engine SMD format but I shall be experting both SMD and BVH versions of each capture, in case something is awry with the SMD one I can use SoftImage XSI to process the BVH into shape instead.

 

Notes

Currently the software cannot do head tracking – that’s why the head doesn’t move.