However, using real image data can raise practical and ethical concerns: The images could run afoul of copyright laws, violate people’s privacy, or be biased against a certain racial or ethnic group. To avoid these pitfalls, researchers can use image generation programs to create synthetic data for model training. But these techniques are limited because expert knowledge is often needed to hand-design an image generation program that can create effective training data.

Researchers from MIT, the MIT-IBM Watson AI Lab, and elsewhere took a different approach. Instead of designing customized image generation programs for a particular training task, they gathered a dataset of 21,000 publicly available programs from the internet. Then they used this large collection of basic image generation programs to train a computer vision model.

These programs produce diverse images that display simple colors and textures. The researchers didn’t curate or alter the programs, which each comprised just a few lines of code.

The models they trained with this large dataset of programs classified images more accurately than other synthetically trained models. And, while their models underperformed those trained with real data, the researchers showed that increasing the number of image programs in the dataset also increased model performance, revealing a path to attaining higher accuracy.

“It turns out that using lots of programs that are uncurated is actually better than using a small set of programs that people need to manipulate. Data are important, but we have shown that you can go pretty far without real data,” says Manel Baradad, an electrical engineering and computer science (EECS) graduate student working in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and lead author of the paper describing this technique.

Co-authors include Tongzhou Wang, an EECS grad student in CSAIL; Rogerio Feris, principal scientist and manager at the MIT-IBM Watson AI Lab; Antonio Torralba, the Delta Electronics Professor of Electrical Engineering and Computer Science and a member of CSAIL; and senior author Phillip Isola, an associate professor in EECS and CSAIL; along with others at JPMorgan Chase Bank and Xyla, Inc. The research will be presented at the Conference on Neural Information Processing Systems.

Rethinking pretraining

Machine-learning models are typically pretrained, which means they are trained on one dataset first to help them build parameters that can be used to tackle a different task. A model for classifying X-rays might be pretrained using a huge dataset of synthetically generated images before it is trained for its actual task using a much smaller dataset of real X-rays.

These researchers previously showed that they could use a handful of image generation programs to create synthetic data for model pretraining, but the programs needed to be carefully designed so the synthetic images matched up with certain properties of real images. This made the technique difficult to scale up.

In the new work, they used an enormous dataset of uncurated image generation programs instead.

They began by gathering a collection of 21,000 images generation programs from the internet. All the programs are written in a simple programming language and comprise just a few snippets of code, so they generate images rapidly.

“These programs have been designed by developers all over the world to produce images that have some of the properties we are interested in. They produce images that look kind of like abstract art,” Baradad explains.

These simple programs can run so quickly that the researchers didn’t need to produce images in advance to train the model. The researchers found they could generate images and train the model simultaneously, which streamlines the process.

They used their massive dataset of image generation programs to pretrain computer vision models for both supervised and unsupervised image classification tasks. In supervised learning, the image data are labeled, while in unsupervised learning the model learns to categorize images without labels.

Improving accuracy

When they compared their pretrained models to state-of-the-art computer vision models that had been pretrained using synthetic data, their models were more accurate, meaning they put images into the correct categories more often. While the accuracy levels were still less than models trained on real data, their technique narrowed the performance gap between models trained on real data and those trained on synthetic data by 38 percent.

“Importantly, we show that for the number of programs you collect, performance scales logarithmically. We do not saturate performance, so if we collect more programs, the model would perform even better. So, there is a way to extend our approach,” Manel says.

The researchers also used each individual image generation program for pretraining, in an effort to uncover factors that contribute to model accuracy. They found that when a program generates a more diverse set of images, the model performs better. They also found that colorful images with scenes that fill the entire canvas tend to improve model performance the most.

Now that they have demonstrated the success of this pretraining approach, the researchers want to extend their technique to other types of data, such as multimodal data that include text and images. They also want to continue exploring ways to improve image classification performance.

“There is still a gap to close with models trained on real data. This gives our research a direction that we hope others will follow,” he says.

Academics from the Universities of Cambridge and Manchester analysed the well-being and self-esteem of more than 11,000 young people from across the UK, using data collected when they were 11, and again when they were 14. The adolescents’ overall ‘subjective well-being’ — their satisfaction with different aspects of life (such as friends, school and family) — dropped significantly during the intervening years.

It is widely accepted that young people’s well-being and mental health are influenced by factors such as economic circumstances and family life. The research shows that notwithstanding this, well-being tends to fall steeply and across the board during early adolescence.

That decline is probably linked to the transition to secondary school at age 11. The study identified that the particular aspects of well-being which changed in early adolescence were typically related to school and peer relationships, suggesting a close connection with shifts in these young people’s academic and social lives.

In addition, students with higher self-esteem at age 11 experienced a less significant drop in well-being at age 14. This indicates that structured efforts to strengthen adolescents’ self-esteem, particularly during the first years of secondary school, could mitigate the likely downturn in well-being and life satisfaction.

Ioannis Katsantonis, a doctoral researcher at the Faculty of Education, University of Cambridge, who led the study said: “Even though this was a large, diverse group of adolescents, we saw a consistent fall in well-being. One of the most striking aspects was the clear association with changes at school. It suggests we urgently need to do more to support students’ well-being at secondary schools across the UK.”

Ros McLellan, Associate Professor at the University of Cambridge, specialist in student well-being, and co-author, said: “The link between self-esteem and well-being seems especially important. Supporting students’ capacity to feel positive about themselves during early adolescence is not a fix-all solution, but it could be highly beneficial, given that we know their well-being is vulnerable.”

Globally, adolescent well-being is in decline. In the UK, the Children’s Society has shown that 12% of young people aged 10 to 17 have poor well-being. Dr Jose Marquez, a Research Associate at the Institute of Education, University of Manchester, and co-author, said: “Until now, we haven’t fully understood how universally poor well-being is experienced. The relationship between well-being and self-esteem has also been unclear.”

The researchers used data from the Millennium Cohort Study, which involves a nationally representative sample of people born between 2000 and 2002 and incorporates standard questionnaires about well-being and self-esteem. They then calculated a well-being ‘score’ for each student, balanced to control for other factors that influence well-being — such as economic advantage, bullying, and general feelings of safety.

While most adolescents were satisfied with life at age 11, the majority were extremely dissatisfied by age 14. By that age, the well-being scores of 79% of the participants fell below what had been the average score for the entire group three years earlier. “This is a statistically significant drop,” Katsantonis said. “It goes far beyond anything we would classify as moderate.”

The study also captured information about the adolescents’ satisfaction with specific aspects of their lives, such as schoolwork, personal appearance, family and friends. This suggested that the most dramatic downturns between 11 and 14 were probably related to school and relationships with peers.

Despite the overall fall, students with better well-being at age 14 tended to be those who had higher self-esteem at age 11. The pattern did not apply in reverse, however: better well-being at age 11 did not predict better self-esteem later. This implies a causal link in which self-esteem seems to protect adolescents from what would otherwise be sharper declines in well-being.

“Supporting self-esteem is not the only thing we need to do to improve young people’s well-being,” Katsantonis said. “It should never, for example, become an excuse not to tackle poverty or address bullying — but it can be used to improve young people’s life satisfaction at this critical stage.”

The researchers identify various ways in which schools could support this. At a basic level, Katsantonis suggested that celebrating students’ achievements, underlining the value of things they had done well, and avoiding negative comparisons with other students, could all help.

More strategically, the study suggests incorporating more features that promote self-esteem into England’s well-being curriculum, and stresses the need to ensure that similar efforts are made across the UK. Recent studies have, for example, highlighted the potential benefits of mindfulness training in schools, and of ‘positive psychology’ initiatives which teach adolescents to set achievable personal goals, and to acknowledge and reflect on their own character strengths.

McLellan added: “It’s really important that this is sustained — it can’t just be a case of doing something once when students start secondary school, or implementing the odd practice here and there. A concerted effort to improve students’ sense of self-worth could have really positive results. Many good teachers are doing this already, but it is perhaps even more important than we thought.”