White Paper: Low Cost Customized Digital Microscope

Kinds of microscopes are running in TE including the stereo microscope, digital microscope, metallographic microscope, polarizing microscope, 3D microscope and electron microscope, etc. They are not only used in the lab, but also in assembly, stamping and molding production lines. The most widely used are stereo microscope (Fig. 1 (a)) and digital microscope (Fig. 1 (b)), both of which belong to optical microscope, which generates the image based on visible light. We can make use of optical microscope for quality inspection, manufacturing assistance, process monitoring and more.

Stereo microscope and digital microscope have their disadvantages. Firstly, they are standard products, which can’t be customized. However, the customers would like to have the configurable device with adjustable field of view, working distance, operation method, operation station, and the software should be customized per various applications. Secondly, bad ergonomic design causes the fatigue and discomfort for long-term use. For example, the stereo microscope will cause the eye fatigue. For digital microscope, the screen is so near that the operator will feel aggressive from the screen. Thirdly, high cost for the optical microscope. The general price of the digital microscope as shown in Fig. 1 (b) is about US$3,350, if we would like to customize some functions, the price will be very high.

One customized digital microscope is necessary to fill the gap between customer’s need and the products in the market. Several challenges are confronted for the development:
1) Low cost for machine vision system with competitive performance: we need to develop one machine vision system including the industrial high-resolution camera, the configurable lens, the lighting unit, the high-resolution display and the ergonomic frame with low cost
2) Ergonomic design: for some applications, the operators will use the microscope continuously for a long time, so the ergonomics of the operation is very important
3) Industrial-level quality to run in the production line: the low cost should not degrade the quality of the product
4) Have the potential to support digital factory: digital factory is important for TE, and many customers are interested in it, so we should consider the potential to connect to network and exchange information
We developed one low cost customized digital microscope. To reduce the cost of the system, we select the embedded system as the processing center, and build the vison system based on the component level. With configurable lens, the precision can achieve 1.9 microns for standard lens and less than 1.0 micron for specific lens. Both working distance and field of view are adjustable. Our ergonomic design with universal frame and 11.6” 1080p high resolution display is suitable for continuous long-term use. To support digital factory, right now, we provide the PC software to connect the digital microscope. Compared with the product (US$ 3,350) in the market, the reference cost of our system is US$ 1,500. Our product passed CE certificate, and are under test in several plants, and many plants show great interests and are inquiring on the product.

We provide two types of customized digital microscope as shown in Fig. 2 and Fig. 3. Both are consisted of the vision system and the frame. The vision system consists of 6-mega-pixel colorful camera to capture the image, the embedded system to process the image, the configurable lens, the LED ring illumination and 11.6” 1080P display. The frame includes the screen arm to adjust the position and angle of the screen, the camera arm to adjust the working distance and angle of the camera, and the base to hold all the components. The difference between type 1 and type 2 product is that type 1 uses the 7 DoF (degree of freedom) screen arm, so it’s more flexible to achieve any angle and any position. The 5 DoF screen arm is more stable. The customer can select the corresponding type based on their applications.
As one digital microscope, the basic performance is the imaging effect which is decided by the whole vision system including the lighting, the camera, the lens and the imaging processing algorithm. We selected several challenging tasks. The first one is the button sensor with about 5 mm diameter. And the challenge is that we need to illuminate such small product at about 110 mm far away. The 110 mm distance is left for in-line hot-tip soldering. The second one is the Micro-fuse product which is about 3 mm width and the surface is covered by the glue. The third one is the soldering product the process of which is very common in TE. Fig. 4 shows the imaging effect on these products, which is very good and you can even see the texture in the surface.

Besides the imaging effect, the operation of the digital microscope is also very important, and this is the most relevant part for operator. Our product is totally optimized for ergonomic operation. The parfocal lens supports optical magnification by turning the regulator as shown in Fig. 5 (a). The position and angle of the screen is adjustable by the 5/7 DoF screen arm. The User Interfaces (UI) of both embedded software and PC software are user-friendly and intuitive for use as shown in Fig. 5 (c) & (d). Our PC software is able to exchange information with digital microscope, for example, save image in PC.

Fig. 6 shows the revolution and architecture of the digital microscope. We used to have several versions. The initial idea is based on the smart camera; we reject this design for cost issue. Then we would like to use the surveillance camera. Based on our test, the quality of surveillance camera is not good enough because it has the video compressing inside which degrades the quality. Also this camera is difficult to be customized. Then we would like to use the concept based on Full PC, but it’s difficult to integrate the full PC into the digital microscope. Finally, we selected the embedded system. The center of digital microscope is one embedded system based on Customized Linux System. The raw data of the camera is processed by embedded system and displayed in the panel. Many image processing algorithms are running in the embedded system, including the Bayer Interpolation, de-noise, sharpness, gamma correction, contrast control, de-color,image enhancement and more. Considering the big data volume, the computation ability of embedded system should be strong enough. Here we use the GPU acceleration, and we will introduce it later.

The digital microscope can communicate with PC through the Ethernet. If needed, the PC can connect with MES/ERP system to distribute the information in the TE internal network. Right now, we are discussing with digital factory team on what information and functions are needed. If the functions are defined, we have the potential to directly communicate with MES/ERP system through embedded system.

For the colorful image, each pixel is consisted of three color channels, R channel, G channel and B channel. However, the photosensor of the camera is sensitive to all visible spectrum, which means normally, it only generates the monochrome image. To generate the colorful image, the color filter will be placed in front of the photosensor to just pass the required color. Then the raw data of the colorful image is one arrangement of RGB pixel as shown in Fig. 7 (a).

Each pixel only contains part of color information; we need to recover the colorful image from this arrangement. There are several recovery algorithms. Basically, the adjacent information of each pixel is highly related to true value of the missing color, our methods is based on the combination of the adjacent pixel. There are two kinds of pattern in Fig. 7 (b), if the green channel exists as shown, just perform average of the adjacent value, or else, we need to consider the influence of the adjacent pixel.

Here I just list the case with R channel, we can also replace the R channel with B channel, and use the equation (2) with same method.

Noise is unavoidable for one image. Here, we analyzed the sources of the noise is the CCD and the environment. This kind of noise are mainly white noise, so we preferred to use the inverse harmonic mean filter. The effect of sharpness can be seen in Fig. 8, less noise is shown in the image after denoise algorithm.

Our sharpness method is based on Kirsch algorithm which calculates the gradient of each points and enhance the region with big amplitude of the gradient, and suppress the region with small amplitude of gradient. The equation can be expressed below. In real calculation, we use the Kirsch template to calculate the gradient at each pixel. The effect of sharpness can be seen in Fig. 9

For 6-mega-pixel colorful camera, the data volume is about 180MB per second, in order to process such big volume data, we need the strong real-time computation ability. Here we use the parallel computing. Just like his name, the parallel computing divides the task into many small jobs, and process these jobs in parallel. The parallel computing takes no burden on the CPU, it’s totally processed by the GPU, so it’s especially designed for big data volume processing as shown in Fig. 10 (a).
The reason we can apply the parallel computing in the image processing algorithm of digital microscope is that the process of our image processing algorithms such as Bayer interpolation, 2D denoise, sharpness and more are only related to the original image, which means the processing of each pixel in the image is uncorrelated, so we can process these pixels in parallel. As shown in Fig. 10 (b), we can process the top-left pixel and bottom-right pixel at the same time.

For digital factory, the base is the communication. Here we design one robust Ethernet connection. Fig. 11 illustrates the work flow of this robust Ethernet connection. The server is listening to the new request from client, if one request arrives, it will create one new communication thread for client. If one connection is established, the system will start the self-diagnosis to check the connection status, and catch each error and recover automatically. The advantages of Ethernet module are： 1) Automatic connection; 2) Automatic catching the error; 3) Automatic recover from the error; 4) Multithread programming without blocking the operation of the software during sending/receiving the file

This paper introduces the innovations about low cost customized digital microscope, including the platform overview, the imaging effect, the ergonomic operation, the system architecture, Bayer interpolation method, 2D Denoise method, image sharpness method. We also introduce how to process big volume data based on parallel computing. Finally, we introduced the robust Ethernet connection.
To reduce the cost of the system, we select the embedded system as the processing center, and build the vison system based on the component level. With configurable lens, the precision can achieve 1.9 microns for standard lens and less than 1.0 micron for specific lens. Both working distance and field of view are adjustable. We support both optical magnification and electrical magnification. Our ergonomic design with universal fixture and 11.6” 1080p high resolution display is suitable for continuous long-term use. Compared with the product (US$ 3,350) in the market, the reference cost of our system is US$ 1,500.
The advantages of this product can be generalized as low cost with industrial level quality, customized for TE application and ergonomic design for continuous long-term use.
To support digital factory, right now, we provide the PC software to connect the digital microscope. The image can be directly saved in PC, and we are also discussing with digital factory team on what functions is necessary to fully realize the digital factory. One customer inspires us to use the tiny PC to replace the embedded system, and we believe it’s one good direction.
Our product passed CE certificate, and is under test in several plants, and many plants show great interests and are inquiring on the product.

We are sincerely thankful to Mr. Josef Sinder who shared lots of information and technology on embedded machine vision system.