One of the state-of-the-art NAS algorithms recently developed by Google took 48,000 hours of work by a squad of graphical processing units (GPUs) to produce a single convolutional neural network, used for image classification and identification tasks. Google has the wherewithal to run hundreds of GPUs and other specialized circuits in parallel, but that’s out of reach for many others.

In a paper being presented at the International Conference on Learning Representations in May, MIT researchers describe an NAS algorithm that can directly learn specialized convolutional neural networks (CNNs) for target hardware platforms — when run on a massive image dataset — in only 200 GPU hours, which could enable far broader use of these types of algorithms.

Resource-strapped researchers and companies could benefit from the time- and cost-saving algorithm, the researchers say. The broad goal is “to democratize AI,” says co-author Song Han, an assistant professor of electrical engineering and computer science and a researcher in the Microsystems Technology Laboratories at MIT. “We want to enable both AI experts and nonexperts to efficiently design neural network architectures with a push-button solution that runs fast on a specific hardware.”

Han adds that such NAS algorithms will never replace human engineers. “The aim is to offload the repetitive and tedious work that comes with designing and refining neural network architectures,” says Han, who is joined on the paper by two researchers in his group, Han Cai and Ligeng Zhu.

“Path-level” binarization and pruning

In their work, the researchers developed ways to delete unnecessary neural network design components, to cut computing times and use only a fraction of hardware memory to run a NAS algorithm. An additional innovation ensures each outputted CNN runs more efficiently on specific hardware platforms — CPUs, GPUs, and mobile devices — than those designed by traditional approaches. In tests, the researchers’ CNNs were 1.8 times faster measured on a mobile phone than traditional gold-standard models with similar accuracy.

A CNN’s architecture consists of layers of computation with adjustable parameters, called “filters,” and the possible connections between those filters. Filters process image pixels in grids of squares — such as 3×3, 5×5, or 7×7 — with each filter covering one square. The filters essentially move across the image and combine all the colors of their covered grid of pixels into a single pixel. Different layers may have different-sized filters, and connect to share data in different ways. The output is a condensed image — from the combined information from all the filters — that can be more easily analyzed by a computer.

Because the number of possible architectures to choose from — called the “search space” — is so large, applying NAS to create a neural network on massive image datasets is computationally prohibitive. Engineers typically run NAS on smaller proxy datasets and transfer their learned CNN architectures to the target task. This generalization method reduces the model’s accuracy, however. Moreover, the same outputted architecture also is applied to all hardware platforms, which leads to efficiency issues.

The researchers trained and tested their new NAS algorithm on an image classification task in the ImageNet dataset, which contains millions of images in a thousand classes. They first created a search space that contains all possible candidate CNN “paths” — meaning how the layers and filters connect to process the data. This gives the NAS algorithm free reign to find an optimal architecture.

This would typically mean all possible paths must be stored in memory, which would exceed GPU memory limits. To address this, the researchers leverage a technique called “path-level binarization,” which stores only one sampled path at a time and saves an order of magnitude in memory consumption. They combine this binarization with “path-level pruning,” a technique that traditionally learns which “neurons” in a neural network can be deleted without affecting the output. Instead of discarding neurons, however, the researchers’ NAS algorithm prunes entire paths, which completely changes the neural network’s architecture.

In training, all paths are initially given the same probability for selection. The algorithm then traces the paths — storing only one at a time — to note the accuracy and loss (a numerical penalty assigned for incorrect predictions) of their outputs. It then adjusts the probabilities of the paths to optimize both accuracy and efficiency. In the end, the algorithm prunes away all the low-probability paths and keeps only the path with the highest probability — which is the final CNN architecture.

Hardware-aware

Another key innovation was making the NAS algorithm “hardware aware,” Han says, meaning it uses the latency on each hardware platform as a feedback signal to optimize the architecture. To measure this latency on mobile devices, for instance, big companies such as Google will employ a “farm” of mobile devices, which is very expensive. The researchers instead built a model that predicts the latency using only a single mobile phone.

For each chosen layer of the network, the algorithm samples the architecture on that latency-prediction model. It then uses that information to design an architecture that runs as quickly as possible, while achieving high accuracy. In experiments, the researchers’ CNN ran nearly twice as fast as a gold-standard model on mobile devices.

One interesting result, Han says, was that their NAS algorithm designed CNN architectures that were long dismissed as being too inefficient — but, in the researchers’ tests, they were actually optimized for certain hardware. For instance, engineers have essentially stopped using 7×7 filters, because they’re computationally more expensive than multiple, smaller filters. Yet, the researchers’ NAS algorithm found architectures with some layers of 7×7 filters ran optimally on GPUs. That’s because GPUs have high parallelization — meaning they compute many calculations simultaneously — so can process a single large filter at once more efficiently than processing multiple small filters one at a time.

“This goes against previous human thinking,” Han says. “The larger the search space, the more unknown things you can find. You don’t know if something will be better than the past human experience. Let the AI figure it out.”

The new data is derived from the STAR (Solenoidal Tracker At RHIC) experiment that added detail — and complexity — to an intriguing puzzle that scientists have been pondering: How the building blocks that make up a proton effect its spin (both protons and neutrons form the center of each atom). The spinning is as essential to a particle’s identity as its electric charge and mass. Because particles have spin, they can act like tiny magnets with a particular polarity. For example, aligning and flipping the polarity of proton spin is the basis for technologies such as magnetic resonance imaging (MRI), an indispensable medical diagnostic tool.

Understanding how this proton spin arises from each proton’s inner building blocks — which include quarks, gluons and what is called a sea of quark-antiquark pairs — may help scientists better understand the nuclear structure of atoms.

“After multiple years of experimental work at RHIC, this exciting new result provides a substantially deeper understanding of fundamental quantum physics questions regarding how quantum fluctuations affect the dynamics of quarks inside the proton,” says Professor Bernd Surrow, vice chair of the Temple Physics Department. “It contributes to our exploration of the structure and dynamics of the visible world around us, including the human body, a table or a car, and what gives them mass in terms of their atomic structures.”

In a field dominated by males, Surrow notes that two women who formerly were graduate students in his laboratory — Amani Kraishan and Devika Gunarathne — were two of the three graduate students who analyzed the data that led to the publication of the research, and continue in the field. Kraishan is currently an adjunct professor both in Temple’s Physics Department and at Al-Hussein Bin Talal University in Jordan. She continues to collaborate with the Temple group under the leadership of Professor Surrow, focusing on the ongoing research program at Brookhaven National Laboratory and a future U.S. Electron-Ion Collider facility.

The third student involved, Jinlong Zhang of Shandong University, subsequently was a postdoctoral fellow at the Lawrence Berkeley National Laboratory and is now at Stony Brook University.

The research was funded by an ongoing grant from the U.S. Department of Energy (DOE). Future goals of the study include more precise measurements of the quarks and antiquarks, and the gluons which “glue” or bind the quarks and antiquarks together to form both protons and neutrons.

Story Source:

Materials provided by Temple University College of Science and Technology. Note: Content may be edited for style and length.

“An increasing number of observational studies have raised awareness of the association between consuming sugary drinks, obesity and the risk of colorectal cancer,” said co-corresponding author Dr. Jihye Yun, assistant professor of molecular and human genetics at Baylor. “The current thought is that sugar is harmful to our health mainly because consuming too much can lead to obesity. We know that obesity increases the risk of many types of cancer including colorectal cancer; however, we were uncertain whether a direct and causal link existed between sugar consumption and cancer. Therefore, I decided to address this important question when I was a postdoc in the Dr. Lewis Cantley lab at Weill Cornell Medicine.”

First, Yun and her colleagues generated a mouse model of early-stage colon cancer where APC gene is deleted. “APC is a gatekeeper in colorectal cancer. Deleting this protein is like removing the breaks of a car. Without it, normal intestinal cells neither stop growing nor die, forming early stage tumors called polyps. More than 90 percent of colorectal cancer patients have this type of APC mutation,” Yun said.

Using this mouse model of the disease, the team tested the effect of consuming sugar-sweetened water on tumor development. The sweetened water was 25 percent high-fructose corn syrup, which is the main sweetener of sugary drinks people consume. High-fructose corn syrup consists of glucose and fructose at a 45:55 ratio.

When the researchers provided the sugary drink in the water bottle for the APC-model mice to drink at their will, mice rapidly gained weight in a month. To prevent the mice from being obese and mimic humans’ daily consumption of one can of soda, the researchers gave the mice a moderate amount of sugary water orally with a special syringe once a day. After two months, the APC-model mice receiving sugary water did not become obese, but developed tumors that were larger and of higher-grade than those in model mice treated with regular water.

“These results suggest that when the animals have early stage of tumors in the intestines — which can occur in many young adult humans by chance and without notice — consuming even modest amounts of high-fructose corn syrup in liquid form can boost tumor growth and progression independently of obesity,” Yun said. “Further research is needed to translate these discovery to people; however, our findings in animal models suggest that chronic consumption of sugary drinks can shorten the time it takes cancer to develop. In humans, it usually takes 20 to 30 years for colorectal cancer to grow from early stage benign tumors to aggressive cancers.”

“This observation in animal models might explain why increased consumption of sweet drinks and other foods with high sugar content over the past 30 years is correlating with an increase in colorectal cancers in 25 to 50-year-olds in the United States,” said Cantley, co-corresponding author, former mentor of Yun and professor of cancer biology in medicine and director of the Sandra and Edward Meyer Cancer Center at Weill Cornell Medicine.

The team then investigated the mechanism by which this sugar promoted tumor growth. They discovered that the APC-model mice receiving modest high-fructose corn syrup had high amounts of fructose in their colons. “We observed that sugary drinks increased the levels of fructose and glucose in the colon and blood, respectively and that tumors could efficiently take up both fructose and glucose via different routes.”

Using cutting-edge technologies to trace the fate of glucose and fructose in tumor tissues, the team showed that fructose was first chemically changed and this process then enabled it to efficiently promote the production of fatty acids, which ultimately contribute to tumor growth.

“Most previous studies used either glucose or fructose alone to study the effect of sugar in animals or cell lines. We thought that this approach did not reflect how people actually consume sugary drinks because neither drinks nor foods have only glucose or fructose. They have both glucose and fructose together in similar amounts,” Yun said. “Our findings suggest that the role of fructose in tumors is to enhance glucose’s role of directing fatty acids synthesis. The resulting abundance of fatty acids can be potentially used by cancer cells to form cellular membranes and signaling molecules, to grow or to influence inflammation.”

To determine whether fructose metabolism or increased fatty acid production was responsible for sugar-induced tumor growth, the researchers modified APC-model mice to lack genes coding for enzymes involved in either fructose metabolism or fatty acid synthesis. One group of APC-model mice lacked an enzyme KHK, which is involved in fructose metabolism, and another group lacked enzyme FASN, which participates in fatty acid synthesis. They found that mice lacking either of these genes did not develop larger tumors, unlike APC-model mice, when fed the same modest amounts of high-fructose corn syrup.

“This study revealed the surprising result that colorectal cancers utilize high-fructose corn syrup, the major ingredient in most sugary sodas and many other processed foods, as a fuel to increase rates of tumor growth,” Cantley said. “While many studies have correlated increased rates of colorectal cancer with diet, this study shows a direct molecular mechanism for the correlation between consumption of sugar and colorectal cancer.”

“Our findings also open new possibilities for treatment,” Yun said. “Unlike glucose, fructose is not essential for the survival and growth of normal cells, which suggests that therapies targeting fructose metabolism are worth exploring. Alternatively, avoiding consuming sugary drinks as much as possible instead of relying on drugs would significantly reduce the availability of sugar in the colon.”

While further studies in humans are necessary, Yun and colleagues hope this research will help to raise public awareness about the potentially harmful consequences consuming sugary drinks has on human health and contribute to reducing the risk and mortality of colorectal cancer worldwide.

Other contributors to this work include Drs. Sukjin Yang, Yumei Wang and Justin Van Riper with Baylor, Marcus Goncalves (lead author), Changyuan Lu, Jordan Trautner, Travis Hartman, Seo-Kyoung Hwang, Charles Murphy, Roxanne Morris, Sam Taylor, Quiying Chen, Steven Gross and Kyu Rhee, all with Weill Cornell Medicine, Chantal Pauli with the University Hospital Zurich, Kaitlyn Bosch with the Icahn School of Medicine at Mount Sinai, H Carl Lekaye with Memorial Sloan Kettering Cancer Center, Jatin Roper with Duke University and Young Kim with Chonnam National University.

This study was supported by the National Institutes of Health, Stand Up 2 Cancer, the Cancer Prevention and Research Institute of Texas and the National Cancer Institute.