Rutvik Desai, University of South Carolina

Generative AI systems like large language models and text-to-image generators can pass rigorous exams that are required of anyone seeking to become a doctor or a lawyer. They can perform better than most people in Mathematical Olympiads. They can write halfway decent poetry, generate aesthetically pleasing paintings and compose original music.

These remarkable capabilities may make it seem like generative artificial intelligence systems are poised to take over human jobs and have a major impact on almost all aspects of society. Yet while the quality of their output sometimes rivals work done by humans, they are also prone to confidently churning out factually incorrect information. Skeptics have also called into question their ability to reason.

Large language models have been built to mimic human language and thinking, but they are far from human. From infancy, human beings learn through countless sensory experiences and interactions with the world around them. Large language models do not learn as humans do – they are instead trained on vast troves of data, most of which is drawn from the internet.

The capabilities of these models are very impressive, and there are AI agents that can attend meetings for you, shop for you or handle insurance claims. But before handing over the keys to a large language model on any important task, it is important to assess how their understanding of the world compares to that of humans.

I’m a researcher who studies language and meaning. My research group developed a novel benchmark that can help people understand the limitations of large language models in understanding meaning.

Making sense of simple word combinations

So what “makes sense” to large language models? Our test involves judging the meaningfulness of two-word noun-noun phrases. For most people who speak fluent English, noun-noun word pairs like “beach ball” and “apple cake” are meaningful, but “ball beach” and “cake apple” have no commonly understood meaning. The reasons for this have nothing to do with grammar. These are phrases that people have come to learn and commonly accept as meaningful, by speaking and interacting with one another over time.

We wanted to see if a large language model had the same sense of meaning of word combinations, so we built a test that measured this ability, using noun-noun pairs for which grammar rules would be useless in determining whether a phrase had recognizable meaning. For example, an adjective-noun pair such as “red ball” is meaningful, while reversing it, “ball red,” renders a meaningless word combination.

The benchmark does not ask the large language model what the words mean. Rather, it tests the large language model’s ability to glean meaning from word pairs, without relying on the crutch of simple grammatical logic. The test does not evaluate an objective right answer per se, but judges whether large language models have a similar sense of meaningfulness as people.

We used a collection of 1,789 noun-noun pairs that had been previously evaluated by human raters on a scale of 1, does not make sense at all, to 5, makes complete sense. We eliminated pairs with intermediate ratings so that there would be a clear separation between pairs with high and low levels of meaningfulness.

We then asked state-of-the-art large language models to rate these word pairs in the same way that the human participants from the previous study had been asked to rate them, using identical instructions. The large language models performed poorly. For example, “cake apple” was rated as having low meaningfulness by humans, with an average rating of around 1 on scale of 0 to 4. But all large language models rated it as more meaningful than 95% of humans would do, rating it between 2 and 4. The difference wasn’t as wide for meaningful phrases such as “dog sled,” though there were cases of a large language model giving such phrases lower ratings than 95% of humans as well.

To aid the large language models, we added more examples to the instructions to see if they would benefit from more context on what is considered a highly meaningful versus a not meaningful word pair. While their performance improved slightly, it was still far poorer than that of humans. To make the task easier still, we asked the large language models to make a binary judgment – say yes or no to whether the phrase makes sense – instead of rating the level of meaningfulness on a scale of 0 to 4. Here, the performance improved, with GPT-4 and Claude 3 Opus performing better than others – but they were still well below human performance.

Creative to a fault

The results suggest that large language models do not have the same sense-making capabilities as human beings. It is worth noting that our test relies on a subjective task, where the gold standard is ratings given by people. There is no objectively right answer, unlike typical large language model evaluation benchmarks involving reasoning, planning or code generation.

The low performance was largely driven by the fact that large language models tended to overestimate the degree to which a noun-noun pair qualified as meaningful. They made sense of things that should not make much sense. In a manner of speaking, the models were being too creative. One possible explanation is that the low-meaningfulness word pairs could make sense in some context. A beach covered with balls could be called a “ball beach.” But there is no common usage of this noun-noun combination among English speakers.

If large language models are to partially or completely replace humans in some tasks, they’ll need to be further developed so that they can get better at making sense of the world, in closer alignment with the ways that humans do. When things are unclear, confusing or just plain nonsense – whether due to a mistake or a malicious attack – it’s important for the models to flag that instead of creatively trying to make sense of almost everything.

If an AI agent automatically responding to emails gets a message intended for another user in error, an appropriate response may be, “Sorry, this does not make sense,” rather than a creative interpretation. If someone in a meeting made incomprehensible remarks, we want an agent that attended the meeting to say the comments did not make sense. The agent should say, “This seems to be talking about a different insurance claim” rather than just “claim denied” if details of a claim don’t make sense.

In other words, it’s more important for an AI agent to have a similar sense of meaning and behave like a human would when uncertain, rather than always providing creative interpretations.

Rutvik Desai, Professor of Psychology, University of South Carolina

This article is republished from The Conversation under a Creative Commons license. Read the original article.

Kristina P. Brant, Penn State

On Valentine’s Day 2025, heavy rains started to fall in parts of rural Appalachia. Over the course of a few days, residents in eastern Kentucky watched as river levels rose and surpassed flood levels. Emergency teams conducted over 1,000 water rescues. Hundreds, if not thousands of people were displaced from homes, and entire business districts filled with mud.

For some, it was the third time in just four years that their homes had flooded, and the process of disposing of destroyed furniture, cleaning out the muck and starting anew is beginning again.

Historic floods wiped out businesses and homes in eastern Kentucky in February 2021, July 2022 and now February 2025. An even greater scale of destruction hit eastern Tennessee and western North Carolina in September 2024, when Hurricane Helene’s rainfall and flooding decimated towns and washed out parts of major highways.

Each of these events was considered to be a “thousand-year flood,” with a 1-in-1,000 chance of happening in a given year. Yet they’re happening more often.

The floods have highlighted the resilience of local people to work together for collective survival in rural Appalachia. But they have also exposed the deep vulnerability of communities, many of which are located along creeks at the base of hills and mountains with poor emergency warning systems. As short-term cleanup leads to long-term recovery efforts, residents can face daunting barriers that leave many facing the same flood risks over and over again.

Exposing a housing crisis

For the past nine years, I have been conducting research on rural health and poverty in Appalachia. It’s a complex region often painted in broad brushstrokes that miss the geographic, socioeconomic and ideological diversity it holds.

Appalachia is home to a vibrant culture, a fierce sense of pride and a strong sense of love. But it is also marked by the omnipresent backdrop of a declining coal industry.

There is considerable local inequality that is often overlooked in a region portrayed as one-dimensional. Poverty levels are indeed high. In Perry County, Kentucky, where one of eastern Kentucky’s larger cities, Hazard, is located, nearly 30% of the population lives under the federal poverty line. But the average income of the top 1% of workers in Perry County is nearly US$470,000 – 17 times more than the average income of the remaining 99%.

This income and wealth inequality translates to unequal land ownership – much of eastern Kentucky’s most desirable land remains in the hands of corporations and families with great generational wealth.

When I first moved to eastern Kentucky in 2016, I was struck by the grave lack of affordable, quality housing. I met families paying $200-$300 a month for a small plot to put a mobile home. Others lived in “found housing” – often-distressed properties owned by family members. They had no lease, no equity and no insurance. They had a place to lay one’s head but lacked long-term stability in the event of disagreement or disaster. This reality was rarely acknowledged by local and state governments.

Eastern Kentucky’s 2021 and 2022 floods turned this into a full-blown housing crisis, with 9,000 homes damaged or destroyed in the 2022 flood alone.

“There was no empty housing or empty places for housing,” one resident involved in local flood recovery efforts told me. “It just was complete disaster because people just didn’t have a place to go.”

Most homeowners did not have flood insurance to assist with rebuilding costs. While many applied to the Federal Emergency Management Agency for assistance, the amounts they received often did not go far. The maximum aid for temporary housing assistance and repairs is $42,500, plus up to an additional $42,500 for other needs related to the disaster.

The federal government often provides more aid for rebuilding through block grants directed to local and state governments, but that money requires congressional approval and can take months to years to arrive. Local community coalitions and organizations stepped in to fill these gaps, but they did not necessarily have sufficient donations or resources to help such large numbers of displaced people.

With a dearth of affordable rentals pre-flood, renters who lost their homes had no place to go. And those living in “found housing” that was destroyed were not eligible for federal support for rebuilding.

The sheer level of devastation also posed challenges. One health care professional told me: “In Appalachia, the way it usually works is if you lose your house or something happens, then you go stay with your brother or your mom or your cousin. … But everybody’s mom and brother and cousin also lost their house. There was nowhere to stay.” From her point of view, “our homelessness just skyrocketed.”

The cost of land – social and economic

After the 2022 flood, the Kentucky Department for Local Government earmarked almost $300 million of federal funding to build new, flood-resilient homes in eastern Kentucky. Yet the question of where to build remained. As another resident involved in local flood recovery efforts told me, “You can give us all the money you want; we don’t have any place to build the house.”

It has always been costly and time-intensive to develop land in Appalachia. Available higher ground tends to be located on former strip mines, and these reclaimed lands require careful geotechnical surveying and sometimes structural reinforcements.

If these areas are remote, the costs of running electric, water and other infrastructure services can also be prohibitive. For this reason, for-profit developers have largely avoided many counties in the region. The head of a nonprofit agency explained to me that, because of this, “The markets have broken. … We have no [housing] market.”

There is also some risk involved in attempting to build homes on new land that has not previously been developed. A local government could pay for undeveloped land to be surveyed and prepared for development, with the prospect of reimbursement by the U.S. Department of Housing and Urban Development if housing is successfully built. But if, after the work to prepare the land, it is still too cost-prohibitive to build a profitable house there, the local government would not receive any reimbursement.

Some counties have found success clearing land for large developments on former strip mine sites. But these former coal mining areas can be considerable distances from towns. Without robust public transportation systems, these distances are especially prohibitive for residents who lack reliable personal transportation.

Another barrier is the high prices that both individual and corporate landowners are asking for properties on higher ground.

The scarcity of desirable land available for sale, combined with increasingly urgent demand, has led to prices unaffordable for most. Another resident involved in local flood recovery efforts explained: “If you paid $5,000 for 30 acres 40 years ago, why won’t you sell that for $100,000? Nope, [they want] $1 million.” That makes it increasingly difficult for both individuals and housing developers to purchase land and build.

One reason for this scarcity is the amount of land that is still owned by outside corporate interests. For example, Kentucky River Properties, formerly Kentucky River Coal Corporation, owns over 270,000 acres across seven counties in the region. While this landholding company leases land to coal, timber and gas companies, it and others like it rarely permit residential development.

But not all unused land is owned by corporations. Some of this land is owned by families with deep roots in the region. People’s attachment to a place often makes them want to stay in their communities, even after disasters. But it can also limit the amount of land available for rebuilding. People are often hesitant to sell land that holds deep significance for their families, even if they are not living there themselves.

One health care professional expressed feeling torn between selling or keeping their own family property after the 2022 flood: “We have a significant amount of property on top of a mountain. I wouldn’t want to sell it because my papa came from nothing. … His generation thought owning land was the greatest thing. … And for him to provide his children and his grandchildren and their great-grandchildren a plot of land that he worked and sweat and ultimately died to give us – people want to hold onto that.”

She recognized that land was in great demand but couldn’t bring herself to sell what she owned. In cases like hers, higher grounds are owned locally but still remain unused.

Moving toward higher ground, slowly

Two years after the 2022 flood, major government funding for rebuilding still has not resulted in a significant number of homes. The state has planned seven communities on higher ground in eastern Kentucky that aim to house 665 new homes. As of early 2025, 14 houses had been completed.

Progress on providing housing on higher ground is slow, and the need is great.

In the meantime, when I conducted interviews during the summer and fall of 2024, many of the mobile home communities that were decimated in the 2022 flood had begun to fill back up. These were flood-risk areas, but there was simply no other place to go.

Last week, I watched on Facebook a friend’s live video footage showing the waters creeping up the sides of the mobile homes in one of those very communities that had flooded in 2022. Another of my friends mused: “I don’t know who constructed all this, but they did an unjustly favor by not thinking how close these towns was to the river. Can’t anyone in Frankfort help us, or has it gone too far?”

With hundreds more people now displaced by the most recent flood, the need for homes on higher grounds has only expanded, and the wait continues.

Kristina P. Brant, Assistant Professor of Rural Sociology, Penn State

This article is republished from The Conversation under a Creative Commons license. Read the original article.