Feature Articles—October 2009 Issue
A Vision System for Measuring Abundance and Size ofDeepsea Scallops
National Instruments LabVIEW Helps Overcome Difficulties In Accurately Calculating
Dimensions of Scallop Shells
By Steve Tomanovich
President
Compass Technical Consulting LLC
Rochester, New York
NOAA’s Fisheries Service conducts annual surveys between July and August to determine the abundance and size distribution of deepsea scallops (Placopecten magelanicus) in areas between Cape Hatteras and Georges Bank in North Carolina.
Measurements of scallop shell height constitute the primary observation on the survey, with approximately 125,000 such measurements typically taken.
Over the years, the agency has experimented with different measurement methods, and in 2005, William Kramer, an information technology specialist at NOAA’s Northeast Fisheries Science Center (NEFSC) lab in Woods Hole, Massachusetts, set out to improve the system using machine vision technology. The aim of the new device, called an automated scallop measurement system, was to achieve better accuracy and speed in measurements and reduce the amount of manpower required to ensure that the data was accurately and efficiently collected.
Kramer and Compass Technical Consulting LLC (Rochester, New York) designed a system that moved the scallops across a conveyor belt on board the trawler. The system counted and measured the scallop shells using Austin, Texas-based National Instruments (NI) LabVIEW graphical development software to capture images from a Fairport, New York-based Imaging Solutions Group (ISG) Firewire camera, run image algorithms and output data. The system logged readings to an integrated computer in addition to the ship’s onboard computer system, which logged additional data about the catch.
Challenges to Building a System
A primary challenge to the vision system was the difficulty in system timing and triggering due to the wide range of scallop sizes and geometries.
The system needed to inspect three species: sea scallops, Icelandic scallops and calico scallops. Each scallop ranged from a few millimeters to 200 millimeters in height. Scallop thickness was also variable, ranging from a few millimeters to more than 50 millimeters.
The scallops could be loaded in any orientation and located anywhere across the conveyer belt.
The size, orientation and thickness variations, along with the presence of debris on the conveyer belt, made the use of typical camera-triggering techniques, such as breaking a beam of light or contact switches, nearly impossible.
Another significant challenge stemmed from the variation in color and texture of the scallop shells. Depending on their age and environment, scallop shells may be covered in barnacles or other organic material, making the shellfish appear dark in reflected light. The scallops were typically much lighter on the bottom side than on the top, and they could come through the system with either side facing upward.
Finally, the geometric measurement of shell height presented a challenge due to the difficulty in consistently locating the correct flat edge of a scallop. The height was measured from the bottom of the shell hinge to the furthest point perpendicular to the hinge. To design an algorithm that would replicate this measurement method exactly, the system needed to detect the flat-hinged edge in a way that was repeatable and accurate. This was particularly complicated by the fact that shells could be cracked or have broken tabs in addition to normal shape variations.
Using the Camera as a Trigger
The first problem the developers needed to address was determining how to trigger the camera, considering that scallops vary considerably in thickness, were placed randomly across the belt and were surrounded by debris. An additional concern with regard to performance was any potential triggering device’s exposure to saltwater and salt air. As a solution, the camera itself was used as a trigger: It continuously captured images (at 30 frames per second), tested each image for the presence of a scallop and processed the image if a scallop was detected. To be effective, it was necessary that each scallop only trigger the system once and that the process remain fast to avoid a bottleneck in the overall processing time.
A series of overlapping regions of interest (ROIs) was created to act as separate soft triggers. A simple and fast calculation of average code value taken at each trigger determined if a scallop was in the camera frame. If the value of any trigger dropped below a specified threshold, the system saved the image for further processing. If no trigger value was below the threshold, the system discarded the image and checked the next image. To avoid double counting, a flag was set for each ROI when it was triggered, all subsequent images were ignored until the average value in that ROI rose above the threshold, and the flag was reset.
When a scallop successfully triggered the system, the image was passed to the main processing algorithm designed to filter out unwanted debris, select the correct scallop to process and calculate the scallop height in millimeters.
Calculating Scallop Height
A scallop’s height was defined as the distance from the flat-hinged edge to the apex of the shell’s curve, but the flat edge was not always obvious. Broken shells, biofouled shells or shells with particularly flat sides presented a problem in detecting the correct signal (flat-hinged edge) from the noise (all other flats). For these reasons, shell area was used as the starting point for calculating height and maximizing accuracy and speed. Shell area was a simpler and more stable measurement because it did not rely on the presence of specific physical characteristics in the shell geometry.
Measuring area was a simple task using the vision development module for LabVIEW, which offered functions for detecting, filtering and analyzing particles.
From the scallop shell’s area, the system used the analysis library to calculate the Waddel disk diameter (WDD) of the scallop, which is the diameter of a circle with the same area. Effectively, the shell height was derived by calculating the diameter of a circle with an area equal to the scallop area.
This calculation was tested on a sample set of scallops to see how well its results correlated to measurements made by hand. The data showed that smaller scallops were generally underestimated, while larger scallops were overestimated. One possible explanation for this is the tendency of scallops to grow wider as they get larger. A correction equation was implemented to adjust the final scallop height measurement as a function of the WDD. The equation was a quadratic of the form: scallop height=a (WDD)2 + b (WDD) + c.
The roots a, b and c were derived empirically and implemented into the height calculation. Samples were run through the system and plotted against the known heights. Accuracy at the prototype stage was determined to be acceptable, although it required further testing under more rigorous conditions on board a ship.
System Conditions, Requirements
System specifications required measurement speeds of 1,800 scallops per hour (two seconds per scallop). Scallops ranged from 25 millimeters to 200 millimeters in diameter.
The system was further upgraded to include the ability to withstand direct saltwater spray, a wide temperature range and proper functioning in lighting conditions from bright daylight to complete darkness.
A standard Dorner 2200 series conveyor was modified to include a stainless steel idler roller (since the drive roller was stainless) and a translucent belt and to accept the electroluminescent backlight. The conveyor was mounted on industrial swivel pads for operation on laboratory benches as well as on board ships.
An aluminum shroud was fabricated to house all of the electronic components, the camera and the computer. A wash system was also created, including a spray bar and squeegee, and installed beneath the conveyor to remove debris. The imaging system consisted of an IEEE 1394 monochrome camera (the ISG Lightwise 1.3 megapixel camera) with an eight-millimeter focal-length lens. The camera was chosen for three reasons: speed, resolution and seamless software compatibility with LabVIEW.
Minimizing lens distortion was critical to the application to ensure that a scallop could be read consistently regardless of its position on the belt. The system required a 30-centimeter field of view and a working distance of about 35 centimeters. The eight-millimeter lens provided the required focal length but resulted in distortion that caused the calculated shell height measurement to vary considerably when placed in the center of the belt versus either edge. Correcting the entire image using point-by-point remapping resulted in accurate measurements but was too time consuming.
After finding a “zero distortion” lens, a simple mathematical correction was implemented in the analysis algorithm to correct for scallop height based on the scallop’s location on the belt (a parameter easily extracted from the particle analysis step). No correction was made in the center of the belt, with maximum correction occurring at the edges. This correction algorithm reduced the effect of lens distortion on the scallop heights to an insignificant level.
A Being Seen Technologies (Bridgewater, Massachusetts) electroluminescent backlight was selected for its simplicity, resistance to the elements and flexibility for integration into the existing system. The low-power application made it safe to use in a wet environment and its lack of delicate parts made it hold up against vibration and rough handling. Light output and uniformity were more than adequate, making the task of image analysis easier than with the reflected-light approach. The panel required lamination to protect it from water, but this was fairly straightforward. The panel’s expected lifetime is approximately 3,500 hours.
Further Steps
The system underwent shipboard testing during the 2006 summer survey. NOAA evaluated robustness, repeatability, speed and ease-of-use to determine additional requirements and upgrades for the system. The agency developed a custom user interface to allow for both development and production run modes, including an audible signal indicating a successful read.
Steve Tomanovich has a Master of Science in image science from the Rochester Institute of Technology and a Bachelor of Science in mechanical engineering from Columbia University. He develops custom vision systems and image analysis algorithms for commercial and government applications and recently worked with Image Science Associates LLC (Williamson, New York) to develop an image quality system for the Library of Congress.
Sea Technology is read worldwide in more than 110 countries by management, engineers, scientists and technical personnel working in industry, government and educational research institutions. Readers are involved with oceanographic research, fisheries management, offshore oil and gas exploration and production, undersea defense including antisubmarine warfare, ocean mining and commercial diving.