4.5. Creating a 3D model from photographs without using a calibration mat

Using a calibration mat is the easiest way to estimate the camera orientation and create 3d models from photographs. However in certain situations this is not practical - e.g. the photos have been supplied by someone else or the object is too large to place on a mat (for example a statue or car).

In 3DSOM Pro version 3 we have included an alternative system which relies on user supplied "markers" provided for each image. We have created a simple interface that allows users to place these markers in the images with a minimum of effort. Once the markers are present, camera estimates are automatically calculated and the software can then create a wireframe and textured mesh in the usual way.

Example project:

To see a completed example project, open markers.som project in the Examples/Markers folder in the 3DSOM Pro installation directory (the default directory is C:/Program Files/3D Software Object Modeller Pro/) or the horse.som project in the Examples/Horse folder.

Step 1 - Taking suitable photos

You can use photos taken with a standard digital camera in JPEG or TIFF format. The software will need to extract special EXIF image tags from the image files to determine the camera focal length. Most digital cameras store this information. However if the image has been processed or cropped the information may be missing (do not crop your images). If you have kept the zoom setting fixed on your camera and the EXIF tags are missing, you will be given the opportunity to manually enter the values. (In this case, you can also change the manually entered values using the File > Re-enter Camera Settings menu item).

The photos should be taken around the object from around 10 to 20 directions. Make sure there are suitable easily identifiable features in the images. These must be fixed points that you can see in several images such as the corner of a label, sharp corner, centre of a small "blob" or other feature with a precise position. You can rotate the object and keep the camera fixed but if you do so, you will need to mark features that lie on the object and not the surrounding backdrop.

Step 2 - Loading images without a mat

To load the images simply start a new project and select the image files. You may be asked if you wish to calibrate the lens - select "No" if you don't have (or can not capture) suitable lens calibration shots. 3DSOM Pro will first look for the calibration mat in the images. This will fail and you will be shown a dialog asking if 3DSOM Pro should continue looking for the calibration mat. Select "No" and 3DSOM Pro will assume you haven't used a calibration mat in any of the images.

If you haven't taken any photos of your own yet, open the Examples/Markers folder and select all the images.

Step 3 - Define the camera locations using the Edit Markers Window.

To start adding markers you need to select the Edit Markers Window by clicking the Edit Markers Tab. The Edit Markers toolbar provides the tools you will need to define markers in all the images. Each marker identifies a single point in 3D space. You should try and identify points that are visible in at least 2 images. In order to reliably determine the camera location and orientations 3DSOM Pro will need a set of 3 images with at least 10 markers visible in all 3 images.

Workflow overview

The initial workflow is as follows:

1. Drag two images from nearby viewpoints into the left and right side of the split screen.
2. Make sure Select or place button is pressed in the down position.
3. Identify a feature in the left image and click on the image. A red pin appears.
4. Find the same 3D point in the right image and click. Another red pin is shown on the right image.
5. Repeat until you have at least 10 markers positioned (try to make sure the points marked do not lie close to a single plane in 3D space).

The subsequent workflow is as follows:

1. Drag an image you have already selected into the left side of the split screen.
2. Drag an image from a close-by viewpoint into the right side of the split screen.
3. Select an existing marker on the left image by clicking a blue pin. It will be then shown in red.
4. Move the mouse over to the right image and click on the position of the same feature in the right image. The new pin will be shown red.
5. Repeat until at least 10 markers are positioned.

TIP:

When placing markers across a pair of images - make sure that the majority of the marked features in the pair of images do not lie close to a single flat plane. For example if your object is similar to a box in shape, make sure that half the points you mark lie on one face and half lie on a different face.

Step 4 - Placing markers across several images

To start work drag a thumbnail into one side of the split screen. You can change the layout and split the screen from the Marker Tools > Arrange Window menu. You can have 1, 2 or 4 images visible. The split screen allows you to quickly select a marker in one image and then place a marker at the corresponding 3D feature location in the other image.

In the above figure you can see a marker has been placed at the centre of the left eye on a toy dinosaur in 2 images.

The currently selected marker is displayed with a red "pin" and the identifying number for that pin is shown in the palette. All other markers are displayed with blue pins. The blue pins are transparent to avoid cluttering the image but as you move the mouse around, the nearest pin is opaque. Also, the pin closest to the cursor is slightly highlighted, and if the same feature is marked in the other displayed images it will also be highlighted.

Select Zoom mode button to smoothly zoom in or out of an image by left-dragging up or down with the mouse. You can also zoom in larger steps using the mousewheel.

Select Pan mode button to translate a zoomed-in image to view the desired region.

Select Select or place button for placing or selecting markers. When selected left-clicking in an image will create a new marker (not identified yet in other images) or place the selected marker in the image. The cursor indicates if you are about to create a new marker (+) or place an existing marker (=). You can use ctrl drag to zoom the image and pan the image by holding SHIFT key as you left-drag.

Under each viewed image is a panel allowing you to flip between the images. Use < and > buttons to move to the previous or next image. You can hold the button down to quickly run through the images. Use |< button to jump to the first image and>| button to jump to the last image.

TIP:	Quick-zoom: Whilst positioning markers hold the SPACEBAR to temporarily zoom in on the image under the cursor. When you release the SPACEBAR the image view returns to the previous zoom. This feature is useful for accurate placement of markers.

Selected marker:

The "id" or reference number for the currently selected marker is displayed in the Selected Marker field. You can use the up and down arrows next to the number to jog the selected marker. You will see the selected marker (red pin) change in the images.

TIP:	NOTE: The selected marker may not be visible in any of the images you are currently viewing - however it is still selected!

To select a marker simply click on the pin in any image when in "Select or place" mode. The pin will then be shown as red.

Deselect and Deleting:

Click Deselect button to deselect any markers so that you can ensure you will create a new marker when clicking in an image.

The Delete selected button deletes the pin for a marker in the current "active" image (outlined in red). Once all the pins for a marker have been deleted, the marker is effectively no longer in use.

Step 5 - Merging markers

Suppose you identified marker "0" in a number of images and marker "1" in a few other images. You then realised that these are all the same feature (e.g. the center of the left eye). You should "merge" markers 0 and 1 so that they are correctly labelled.

To do this simply click on marker 0 in any image, then click on Merge markers button and then click on marker 1 in any image.

Step 6 - Feedback on camera estimates

As markers are added, 3DSOM Pro will attempt to estimate camera orientations and positions for each image. Cameras are identified into one or more "groups". Within a single group, the relative camera orientations and positions are defined. Each group is displayed in the thumbnail view with a different coloured background (in the screenshot all the thumbnails are in the first group coloured green).

If you have several groupings with different colours then you need to ensure there are more marked features that are present in 2 or more different groups.

Cameras with no group (white backdrop) need more marker features.

Once all the thumbnails have a green background, you have placed enough markers.

Step 7 - Checking results

You can select the View Model Window and display the camera positions using the Model Tools > Display Mode > Display Cameras menu item. The cameras will be displayed in yellow.

In the View Model Window you can also see the 3D points calculated for each marker. Use the Show points only button to display the 3D points without the 3D surface and the Show points and surface button to show the points overlayed on the 3D surface.

You should check that the camera positions look sensible and are positioned around the 3D points (in the configuration you took the photos) and pointing towards the 3D points. If the camera positions are bunched together or appear incorrect, you may need to add more markers.

Step 8 - Continue with model creation

Once you have successfully created the cameras you can continue to create a textured 3d model in the normal manner:

• Mask images: To create a surface you will need to create masks for each image using the Mask tools. If you are using outdoor images without a backdrop you will need to manually paint the mask using the tools provided or via Photoshop. See the section Masking the images.
• Generate Wireframe: Assuming you have views all around the object can you create a triangular mesh using the Surface Wizard and also optimise the mesh to smooth the surface and improve accuracy. See the section Building the wireframe model.
• Generate Texture map: Once you have the surface mesh, create the texture map using the Texture Wizard in the normal way. See the section Adding the surface texture.