Skip to sections.
Kao Corporation’s multi-angle image capturing system is changing the development of cosmetics from a sensory process to a goal-oriented process. The system consists of 20 digital single-lens reflex (SLR) cameras set up around a person in a 3-meter diameter hemispherical installation. It photographs the person’s face from all angles and quantifies the data.
Nikon Systems Inc. is responsible for developing the firmware for the precision instrumentation and the software for image analysis and measurement (the graphics-based software), which enables the coordinated and centralized operation of the multiple digital SLR (D70) cameras. Why is this system necessary and how will it be used? We posed these questions to Dr. Eiichiro Misaki from the First Research Office, Beauty Care Research Center, the Kao Corporation Tokyo Laboratory.
The multi-angle image capturing system
20 digital SLR cameras and 41 lighting devices are fixed to a hemispherical scaffold with a diameter of 3 meters. A subject at the center of the apparatus can be photographed from multiple angles simultaneously.
The development of cosmetics involves the product development department, whose marketing specialists are in touch with the customer, and the research department, clarifying what it is that the customer is looking for. A new product concept is then developed. With the process that has been employed up until now, however, the sought-after image has often been comparatively vague. For example, those developing the concept would look at photographs in fashion magazines and say, “Couldn’t we go for this look?” or “Wouldn’t it be great to look about 10 years younger?” Accordingly, we need the ability to share overall perceptions of skin tone and color, such as its smoothness or moistness, using our sensory images.
Our current area of responsibility, and one of our company’s key products, is foundation for women. We’ve learned that there is a difference of five to ten years between a user’s real age and the age that she feels. In other words, she probably wants to look five to ten years younger. However, subjective requirements for smoothness and moistness do not make for product development based on scientific knowledge. We hoped to be able to communicate between the customer and the developer using a common language, so that we would be able to comment on what was beautiful and what was not, while examining the skin of a 20-year-old woman and the skin of a 30-year-old woman to make the differences visible.
Up until now, research has investigated the physical properties of the skin, for example, by looking at the condition of the keratin in the surface layer of the skin or by measuring skin color. This can be carried out in numerical terms; however, it is hard to understand how these numbers relate to a sensory perception of how beautiful the skin is. In this instance, we simply look at what people regard as beautiful. For example, if shown two scenic photographs, most people will prefer the one that they feel is more beautiful. Likewise, couldn’t we also compare people’s age and the beauty of their skin on a visual basis?
This is how we came to try and actually photograph the skin.
A woman sits in front of a mirror and checks to see that her makeup is just right, tilting her neck and turning to the side. This is because she is seen by other people from many different directions in the course of her day or as she goes about town. Likewise, when we evaluate a product, we also view it from different directions in order to check its texture and color development. An impression of beauty or youth is probably formed from the overall picture of someone, as seen from every angle. This project is designed to create a system to reproduce this phenomenon.
Before the system attained its final form, we tried taking photographs with both a single camera, moving it from one point to the next, and using multiple cameras. Photography is easy when the subject is a stationary object, such as a product package. However, our subject is a human being. A person can keep still—without changing her expression or posture—for about a minute. In the course of our actual research, we photograph ordinary women, and on occasion also photograph small children. Rather than ask people not to move for 30 minutes (an unreasonable request), it was better to arrange a number of cameras and photograph the subject from multiple angles simultaneously.
Also, if someone is forced to keep still in an unrealistic fashion, they will cease to appear natural. People are aware that they appear differently when seen from the right and from the left. Since this system photographs the subject from all sides simultaneously, the process is completed without the subject being given any sense of where she is being photographed from.
Checking the images on the console monitor. The results of shooting photographs from 20 perspectives simultaneously are shown.
A scaffold stands in front of the subject, encircling her. This scaffold is fitted with 20 digital SLR cameras and 41 lighting devices. Since we particularly wanted the cameras to be equidistant from the subject, the apparatus is hemispherical. No camera on the system is below the eye level of the person being photographed. Although cameras could be positioned below eye level, we thought that people have a strong psychological resistance to being photographed from below. Even so, there are lighting devices positioned below eye level.
One novel idea at the design stage was to build a special booth in which all the cameras and light sources would be shrouded. The plan was that the subject would enter the booth and be photographed. Although it would have been easier to design the camera set-up that way, it is probably psychologically easier for the subject if the cameras and light sources are visible to a certain extent. It is disconcerting to know that you are being photographed without being able to see any of the equipment. There are also problems relating to personal information and image rights. When you photograph ordinary people, you must make clear the kind of photograph that you are taking and what you will use it for. Thus, we constructed the system in this manner, in the belief that having everything out in the open gives people a sense of security.
Since there have been considerable advances in digital image processing, there was no question that digital photography would be employed. As regards data resolution, high-resolution data makes processing demanding, due to its high volume. However, reducing image resolution would be problematic when viewing part of the data afterwards. When evaluating cosmetics, we can only get a picture of a woman’s skin at the time it is being photographed. If there is a break between photographs, the condition of the skin changes—the condition of the skin may even change within a single shoot. A standard macro lens is used. After trying lenses of several different focal lengths and checking the results, we finally selected the Micro Nikkor lens.
As we wanted to photograph a person’s face, we thought that a lateral angle of 180 degrees would be just right. For angular analysis, a 90-degree optical angle is often divided into two (at the 45-degree point) and into three (at the 30-degree and 60-degree points). For this reason, 15-degree units (which are applicable to both systems) are often considered. The lens that we are using can properly achieve the desired precision (the ability to photograph a pore) at a range of approximately 1.5 m. Having thus decided on a 3-m-diameter semi-circular configuration, we decided that it was appropriate to space out the light sources and cameras at 15-degree intervals.
Thus, 13 cameras are lined up at face level. There are also three cameras spaced at 15-degree intervals above eye level, facing the subject (these are fewer in number since the subject’s hair will somewhat conceal her face when viewed from front above). A further four cameras are positioned diagonally above the subject, two to the right and two to the left, also spaced at 15-degree intervals. Thus, the subject is photographed by a total of 20 cameras. There is also one more camera, located behind the main apparatus to simultaneously record the photographic conditions when the subject is photographed. All 21 cameras can be operated from a computer via a network of USB cables and hubs.
One reason for selecting an SLR camera was that it features the required precision, as mentioned earlier. A second reason is that it is easy to change the lens, so that if some of the cameras are switched to a lens with a long focal length, it is possible to take both ordinary multi-angle images and enlarged full-face photographs in a single photographic shoot, and thus avoid overtaxing the subject. By opting for SLR cameras, we can retain the freedom in future to carry out zoom photography if need be, or to photograph the entire body, if the opportunity arises.
This was also a major factor in our selection of Nikon products. In view of the importance of changing lenses, we needed to pick a reliable camera manufacturer with a comprehensive line-up of lens products from whom we can expect consistent support in the future.
The architecture of the multi-angle image capturing system
The system connects 21 cameras via USB cables and nine USB hubs, and can be controlled using software on a PC. After the photographs are taken, the data stored on the cameras is automatically transmitted to the server in sequence, and displayed on three monitors (one for the photography log, one for the listing of multi-angle images, and one for the image from the optional camera).
Camera set-up and configuration for the multi-angle image capturing system was carried out by Kao, as there was no discussion at all of the overall concept of a multi-angle image capturing system when the order was placed. Nikon wrote the software for us, and this was in response to an outline request for a system in which 20 cameras take photographs simultaneously and the data is transmitted to a computer and displayed. Even the development of the final user interface went ahead based on an outline specification, with consultation as required. As the system is used for product development, there were aspects that had to remain confidential.
With a single click of the shutter, file names are set for the data for the 21 photographs that allow the position of the relevant camera to be identified and include a time stamp, and the data is placed in a single folder. Details such as the specifications for these operations and the layout of the operations screen were not decided until the end.*1 In addition, we made a number of additional requests to Nikon. For example, since batteries were unnecessary, we asked Nikon to remove them (in order to reduce the weight load on the equipment scaffold), and we asked Nikon to add another camera (to photograph the photographic conditions).
We knew nothing about how cameras work, but when we conducted a trial using one camera, we realized how convenient digital photography is. We believed that when it came to photography from multiple angles, it was probably theoretically possible to trigger the cameras simultaneously and to control the process using a program. We were also confident that Nikon would be up to the task. I had never been that interested in photography and cameras; however, in the course of the project, I developed a passion for them, and came to want a digital SLR camera for myself (laughs).
The difficult part is the experimentation with the equipment, which is essential for acquisition of accurate data. First, we construct equipment to simply check the cameras, and repeatedly check all 20 cameras for errors using the same subject under the same lighting conditions. Here also, Nikon has proved to have been a wise choice. The uniform quality of the cameras is of great benefit. We had been worried about what we should do if we received an item that differed from our request guidelines. Even returning items to Nikon is absolutely no problem—provided we have been using them in the normal fashion (laughs).
For the light used as the light sources, each individual item is inspected before installation and the actual environment is inspected after installation. The thorough operational planning can be seen in details such as the alignment mechanism that uses a laser pointer attached to a strobe shoe and the redundancy in the USB cabling.
In contrast, we threw away about half the lights that were supplied for lighting. We are using halogen lamps that have a color temperature similar to that of sunlight, because we are considering the possibility of using a different light (with a particularly long wavelength characteristic) in future and are trying out mass-market products. In this case too, we are building special equipment, using a spectrometer to check the wavelength characteristics of the lights one by one, and then individually checking them again after installing them in the multi-angle image capturing system. This is because voltage characteristics change with differences in wiring lengths; however, even after adjustments are made to change the voltage of the lighting equipment, there are still some lights whose characteristic values differ from the average values.
At the outset I had no detailed knowledge of electronics or optics—my specialty being powder technology. Thus, although discussions proceeded smoothly during the origination of the project and design stage, actually putting it into practice was hard for me. With optics, we should be able to make estimates based on theory; however, it had never occurred to me that when you actually try to position the lights, electrical values will differ from those observed in prior checks, as a result of phenomena such as voltage drops due to the effect of factors such as cable length and wiring. This seems obvious in hindsight—however, I was really learning on the job.
It would have been easier to assign the troublesome work of precisely adjusting the cameras and the lighting to someone else. However, cost was a factor. Even more importantly, actually doing it myself also had the advantage that I could get a real understanding and feeling for the level of precision that was possible.
In present-day cosmetics foundations that adjust the color of the skin, many substances are used that perform color development by means of thin film interference, among which are those that use structural color development. Micronized pearl-colored raw materials that are used as ingredients in foundation raw materials appear white when scattered, but appear blue or red when the particles are applied to a substrate and lined up in one direction.
Traditionally, the development of cosmetics products has consisted of a sequence of manufacturing trial and error. The method that is used to determine good products that will fulfill the aim of creating a beautiful or young look is to repeatedly search for an ingredient, prescribe and evaluate it, and then try the next one. In the course of this process, a researcher relies on his or her intuition, which cannot be quantified, and there is no substitute for repeated trial and error. Given the requirement to come up with a product of some description within a fixed development period, compromise is inevitable.
If you can quantify an image of beauty or youthful looks, you can use a more intelligent method. For example, suppose the aim is to recreate beautiful skin (to make the customer look 10 years younger). Having set this as the target, the first step is to investigate what it is that changes and what it is that diminishes as someone ages. Then, since we know that the skin’s apparent color differs when seen from certain angles, we will be able to see how ingredients ought best to be used to produce color, so as to enable the cosmetics to make up for the deficiencies.
With the multi-angle image capturing system, we first compare a person with beautiful skin and a person with ordinary skin. Under conditions in which the maximum amount of light is being reflected by the surface of the skin, we can see that the skin’s luster varies. The beautiful skin partially absorbs light, and this brings out its color. For the ordinary skin, however, the color of the lighting is reflected unchanged, and the skin gleams white. Accordingly, we have developed a pearl-colored raw material that performs structural color development and makes up for deficiencies in skin color at certain angles only, in order to reproduce and bring out a brilliant skin color in the areas where the skin gleams with a white luster.
The term “structural color” refers to a color that is perceived due to a microscopic structure in the nanometer to micrometer range (close to the wavelength of light), such as a thin film or a regular pattern of bumps formed on the surface of a material. Light is reflected and bent in a complex fashion, and only specified wavelengths (or colors) are intensified, yielding the color that is seen. It is structural color that causes the light that strikes the recording surface of a compact disc (CD) to appear iridescent, and soap bubbles to appear to be colored. It is not the color of the material itself that is perceived—the perceived color changes as the light’s angle of incidence changes. Hence, when a structural color is used on the skin, it does not appear smooth but has a three-dimensional effect.
For some time, skin color has been adjusted by putting pearl-colored raw materials available on the market in foundation. Now that we understand this mechanism, however, it is easy to design materials. Using the principle of structural color to carry out color development allows us to use simulations to make predictions.
The pearl-colored raw material consists of mica particles covered in a thin film of titanium dioxide (TiO2). The thickness of this film is less than the wavelength of light, and color development results from the interference between the light reflected off the surface of the film and the light reflected from the inner surface of the film.
Since light interference is involved, in theory the thickness of the film and the color development ought to be governed by a mathematical formula. However, the results for simulations and the results for the actual materials were different. This means that the mathematical formula that has been widely used up until now proved to be inadequate. Some adjustments and improvements here brought the angular color characteristics and scattering spectrum into line with the simulation results. However, even this did not yield the brilliant gold color that expresses beautiful skin. Unless materials with a fairly high chroma are used, the effect is insufficient to make up for deficiencies in skin coloration. We achieved this by constructing an intermediate layer between the thin film and the substrate, and adding ferric oxide (Fe2O3) to it. Adding ferric oxide changes the refraction index and causes certain wavelengths to be absorbed. By incorporating all these complex conditions, we discovered from the simulations how to fabricate materials that would yield the brilliant gold color.
The next stage was fabrication of the material; however, our first attempt yielded red color development—not gold. This can presumably be attributed to discrepancies in the simulation. Since we had confidence in the results of the simulation, we speculated that the condition of the layer may differ from the design, and this yielded a substance whose performance matched the results of our calculations. Using this pearl-colored powder has enabled us to create a foundation which carries out structural color development that gives the face a brilliant gold color when viewed from the front, making for a light and beautiful skin color, and creates shades that are redder further round the sides of the face to accentuate its three-dimensional quality.
A schematic diagram of the new pearl-colored material developed on the basis of research using the multi-angle image capturing system, and a structural diagram. Specular reflection causes high chroma, due to the insertion of the nanometer intermediate layer.
Although it is possible that other manufacturers are optimizing parts of the development process, I believe that this is the first attempt to do so in a comprehensive fashion. In general, manufacturers initially develop products using ordinary materials. In Japan, there are only a few cosmetics manufacturers like Kao who manufacture their own materials. The technology used in this foundation represents a unique effort—from material design onwards. The combination of a thin film and an intermediate layer has unlimited possibilities. It would probably have been impossible to realize this using the traditional trial-and-error method.
If you start with existing materials, you are constrained by their strengths and weaknesses throughout the entire process. If a key material is not available on the market, we are forced to make it ourselves. However, I would not want to go through a repeated trial-and-error process at that stage. Since we have an image of our ideal, it is better to work toward creating that ideal from the start.
Our goal is to make good products in the limited time available. In my own field, powder technology, it is said that powders are the devil's playthings, and that you never know what a powder will do until you test it. The philosophy was to engage in trial and error, experiment repeatedly, and wait for good results to emerge. Although this may work in some environments, technology is advancing, nanotechnology is being applied effectively, and the speeds of digital image processing and simulation are increasing. Our current efforts are rooted in the notion that we should combine these technological innovations. As such, from the outset, our goal has been to change the development process for cosmetics.
The control consoles for the multi-angle image capturing system. The controller for the lighting equipment is on the right.
So far, we have been using multi-angle images to see how color and luster change when the light source and perspective are changed. We are now thinking that we could use the multi-angle image capturing system for problem solving, for example, to investigate what the best products for disguising pores would be, or for improving makeup technology. Currently, there are various areas under consideration, including those that have nothing to do with development, and if we can clarify the problem areas, the ultimate objective should become clear. It should then be easy to incorporate these discoveries in the product development process.
Changing the lights and installing filters, enabling research to be conducted under various different lighting conditions, is also a possibility. However, this would be undertaken after the objective had been determined. This means investigating customer requirements and determining product concept. Up until now, the way in which the customer has been approached has also been rather unclear. However, from now on, we will be able to offer the customer objective suggestions, using photographs taken from multiple angles. Furthermore, in test monitoring of users, showing them photographs and listening to them would seem to make it psychologically easier to ascertain their requirements.
Since structural color is an extremely interesting field, there is a constant temptation to try and manufacture something new. However, the technical obstacles are considerable. In the development of cosmetics, only materials that have been verified as safe may be used. Thus, we cannot simply adopt new materials, no matter how good their color development properties are. In the case of structural color, however, we can simply use a combination of raw materials that is known not to be harmful to the skin or the thickness of the thin film to freely control color development. In this regard too, research into structural color will continue to attract interest.
A portion of this site requires Macromedia Flash Player. To install the latest version of this software, start by clicking the button at left for a free download.
Posted March 2008