Can liberal political philosophy support anti-racism?

John Rawls and Philip Pettit agree about the idea that a liberal democracy depends on the idea that all citizens have equal liberties, rights, worth, and dignity. Therefore they also agree that social and legal arrangements that are incompatible with equal rights, equal liberties, and equal dignity are illegitimate. They disagree in some details about what all of this means — Pettit refers to liberty as “the absence of domination” (link), while Rawls emphasizes the liberty to pursue one’s conception of the good in the way he or she chooses (link). But the common ground between these leading advocates of liberal democracy is extensive. And each philosopher provides an unequivocal basis for rejecting mistreatment and discrimination based on race, ethnicity, or other social characteristics.

What is less clear is whether either of these philosophers has a place for the idea that an inclusive multicultural democracy — in Rousseau’s conception, a “free community of equals” — has a positive value for the whole of society, and whether enhancing this value is itself a legitimate function of a democratic state. These are separate questions, and it is possible that Rawls and Pettit would affirm the first but deny the second. Rawls’s view in A Theory of Justice and Political Liberalism is largely that the function of the state is (1) to establish and secure the system of law within which all citizens enjoy maximal, equal rights and liberties, and (2) to establish a legitimate majoritarian process through which policies and laws are enacted subject to the authority of the majority of citizens. Anything more extensive than this falls outside the scope of legitimate exercise of coercive authority, according to Rawls. His distinction between a “political conception of the good” and a “comprehensive conception of the good” is crucial for his view of the scope of the state’s authority: the former consists of the minimal commitments that all citizens share concerning the functions and limitations of the state and its legal system; whereas the latter consists of a fully developed set of ideas and values that individuals or groups may adopt for orienting their lives and activities, but concerning which the liberal state must remain neutral (link).

So, for example, the state cannot undertake on its own, without democratically enacted legislation, to ensure a beautiful natural environment, simply on a governmental judgement that “all of society is better off when the natural environment is maintained for public enjoyment”. For Rawls, this judgment falls within a “comprehensive conception of the good”, and cannot be taken as a function of government without majority-supported legislation. By contrast, the state is fully authorized to enact rules and procedures that guarantee free and unfettered elections, because the right to vote is a fundamental democratic right shared equally by all citizens and part of the “overlapping consensus” (the political conception of the good) required for any democracy.

We can raise this question from two related perspectives: the perspective of idealized political philosophy (abstract theorizing about what constitutes a good and well-ordered society and state) and the perspective of the minimum legal and constitutional requirements needed to ensure the equal liberties and rights of individual citizens (minimalist theory). The first is intended to articulate a vision for the future of social life within a democratic society, while the second perspective is intended to articulate the protections of rights and liberties that every legitimate state must embody.

Is there a basis in the “minimalist” version of the liberal democratic state that gives broad authority to government as well as private and public organizations to take positive measures to cultivate attitudes of racial acceptance and respect among their constituents? Can a liberal democratic state enact a set of arrangements through which citizens will learn the values of tolerance and compassion, and learn of the harmful effects of attitudes involving negative stereotypes about members of other groups? And if the answer is that government itself cannot undertake such measures, are private organizations and relatively autonomous public institutions free to do so when it comes to organizing the functioning, supervision, and training of an agency, a private workplace, or a university? Or does the minimalist perspective on political philosophy involve only restrictions on actions that harm others or reduce the freedoms of others, with no basis for undertaking to change how people think? Perhaps “cultivating mutual tolerance, interest, and respect” is itself a value about which reasonable people may differ.

We might imagine, for example, a “liberal cultural separatist” who fully endorses and respects the equal civil and legal rights of member of other groups, and this person condemns discrimination against individuals based on their particular characteristics. However, this person prefers to associate with members of his/her own group and believes that others should do so as well. “Separation of groups is best for the social order,” according to this version of the good society. If so, then according to Rawls the goal of “building a tolerant and inter-connected society” belongs to a “comprehensive conception of the good”, and the state must remain neutral about this value. Citizens are entirely free to form their own associations and advocacy groups around these values, but the state must not take a side.

To put the point somewhat differently, are the ideas of tolerance, compassion, and respect simply specific visions of inter-group relations, to be debated alongside a number of competing views as “comprehensive conceptions of the good” — with the implication that this particular vision cannot be enforced through state mandates? A traditional liberal like John Stuart Mill (and perhaps Rawls as well) might argue that if the values of a tolerant and respectful society make up a compelling idea, then advocates should be able to persuade a majority of citizens to agree. In that case these programs can be democratically enacted through enabling legislation. But if the idea remains “visionary and confined to a small minority” then the state cannot use its coercive power to enact policies based on this vision of a more tolerant society.

On this line of thought the answer to the question is not much different from the formulation offered by J.S. Mill: liberalism is committed to individual freedoms, including freedom of speech and association, and these commitments are fundamental. So the legal system must give substantial deference to the opinions, statements, and programs of individuals, and much of the same deference is due to private and public institutions as well. The task of struggling for anti-racism, toleration, mutual respect, and communication across major racial and ethnic divisions is to be left to private associations rather than to state legislative authority.

On this view of a liberal society, individuals and their free associations have the right to advocate for an inclusive multicultural democracy, and for the steps needed to create such a world. What about other organizations? Is it legitimate for businesses, public school systems, colleges and universities, and labor unions to adopt similar resolutions? Is it legitimate for these organizations within civil society to enact procedures and requirements within their scope that are designed to influence the thinking and behavior of the individuals who make up those organizations? Do private organizations like businesses, private universities, and non-profit organizations have a zone of autonomy that permits them to undertake “pro-inclusiveness” policies, procedures, and training regimes? And what about organizations owned or directed by the “liberal cultural separatists” considered above — do they too have autonomy to enact processes that further embed prejudice and stereotype?

What seems to be lacking within liberalism, both traditional and contemporary, is a way of coping with “imperfect justice” and the fact that the current generation of society may embody ways of thinking, stereotypes, and prejudices that serve to reproduce racist, sexist, or religious discrimination in the next generation. (Charles Mills explores these ideas in The Racial Contract and Black Rights/White Wrongs: The Critique of Racial Liberalism.) However, to go beyond racism, it seems clear that a process of moral transformation is needed. Children, young people, and adults need to come to understand the history of racist thinking and action in our country and to recognize the value of respecting the equality and dignity of members of other groups.

The assault on “Diversity, Equity, and Inclusion” programs in universities, corporations, and government departments currently underway (link) seems to reflect a status quo mentality when it comes to racial prejudice and stereotype: “we are who we are, and there is no need for change”. More bluntly, it reflects an ideology of white supremacy. But as Dr. Martin Luther King Jr. expressed so vividly, racial equality and justice will only come to pass when the people of our nation have undertaken the hard work of confronting the realities and persistence of racism. Personal transformation is a necessary step on the way to human equality.

These reflections suggest that liberal theories of justice like those offered by Pettit and Rawls need to be supplemented by two things: (1) A clear and developed elaboration of the value of a pluralistic multicultural democracy based on real equality and respect across groups. Such an account will demonstrate both the value of such a society and its connection to profound ideas about liberty, equality, and humanity. (2) A clear account of the extended processes of learning that will be needed to get from here to there. Such an account will incorporate a realistic appraisal of the ways in which persistent racial attitudes and habits inform the next generation’s social environment as well, and it will provide some ideas about how to accomplish these transformations at multiple levels. Achieving racial justice, and ending antagonism and mistrust across groups in society, requires good laws, but it also requires sustained processes of personal transformation for citizens of all ages.

A political philosophy for an inclusive multicultural democracy

 We might say that a political philosophy is a formulation of the normative ideals that the philosopher holds to be primary in implementing the moral and social facts of “assemblages of free individuals in society, with conflicts of interest and belief”. How should such a society be organized? What values should it aspire to realize through its laws and practices? What forms of constitution, law, and state are best for the realization of the individuals who live within the society?

Here I want to lay out the skeleton of a political philosophy incorporating the ideals of an inclusive multicultural democracy. I maintain that a stable and inclusive multicultural democracy is a positive value for the whole of society: all citizens are benefited by a varied and harmonious population of peoples with distinct traditions, values, and practices. This is a society in which there are many groups and identities in society (racial, ethnic, sexual, class, nationality), and in which members of these groups have the moral emotions of compassion and respect towards members of other groups. Difference exists without discrimination and prejudice; more fundamentally, difference exists within the context of a cohesive sense of shared community. Rather than antagonism there is friendship across groups.

This political philosophy corresponds closely to many of the ideas about racial justice formulated by Dr. Martin Luther King, Jr. during his efforts to find a way towards greater racial justice in the United States. Several ideas in King’s political philosophy stand out in particular. First, his idea of the “Beloved Community” emphasizes that the transformations needed to achieve a good multiracial society go beyond establishing formal legal equality for all, to enhancing the moral emotions of compassion and mutual caring as human beings [“Birth of a New Nation” (1957), “Facing the Challenge of a New Age” (1957), “Stride Toward Freedom” (1958)]. The multiracial society that King envisioned involves a cohesive community of mutual respect, understanding, and compassion. Second, King’s discussions of a future for racial justice in America and other countries always invokes the need for moral transformation — for a change in the way that people think about their fellow human beings with compassion and respect. Third, King’s conception of a just future invokes a deep recognition of shared humanity across race, ethnicity, and other group identities. This advances the idea that citizens in a just multicultural democracy will experience the moral emotions of compassion and respect for each other. King described this as a kind of “civic friendship” in which people from different groups succeed in living together harmoniously and leads them to experience a sense of goodwill and shared identity with their fellow citizens.

All of this requires moral transformation of generations of citizens. Freedom from discrimination, violence, and prejudice is only the beginning of a good multiracial society. Rather, real, concrete human beings need to come to think and feel differently about their humanity and the humanity of others around them.

For King these ideas are placed within a religious or spiritual interpretation, but the theological point is not essential; the values King articulates can be equally endorsed from a purely secular and humanist point of view.

What are those good effects thought to follow from achieving a “harmonious, tolerant, and mutually respectful multicultural society”? Several lines of thought are relevant here. Take Martha Nussbaum’s argument that “compassion is the fundamental moral emotion” and her view that compassion must be learned (link). Exposure to the lived experience and traditions of people different from oneself is a powerful way of developing greater compassion for others — and this presumably makes for better neighbors, citizens, and human beings. Or take the idea that “ethnic conflict is a constant threat in divided societies” (link). It can be argued that a multicultural society that has extensive experience in engaging across communities (racial groups, religious groups, ethnic groups, sexual identities) will be more resilient when crises and conflicts arise (link). I’m thinking here of events like disagreements between neighbors that escalate along racial or ethnic lines; occasions when opportunistic political leaders actively cultivate inter-group suspicion and antagonism; or when economic events occur that seem to divide groups along racial, religious, or ethnic lines. A final benefit that we can imagine is the rewarding experience of learning from another person’s experience or traditions about the norms and events that shaped them, and the reflective benefit that such learning can have for one’s understanding of him or herself (link). For reasons like these we might say that we are all better off if we live in a tolerant and respectful community, and the community itself will be healthier and perhaps even more productive as a result.

So achieving a just, stable, and cohesive multicultural democracy is a worthwhile goal. But will a well-ordered liberal democratic state have the authority — and perhaps the duty — to take measures that enhance the workings of a multicultural democracy? These ideas about inclusive democracy make up a feature of a specific political philosophy, a theory of the values that a good and just society ought to strive to realize. But how can these features be realized within a liberal constitution defining “a good and just society”? Would we do better to recognize that these are ideas open to democratic debate for consideration by the population itself?

In order to bring about a “stable and inclusive multicultural democracy”, it seems evident that citizens themselves need to embrace these values. And this means education. This may mean taking steps to change the ways that children, young people, and adults think about the different groups in their society through education, celebrations, and “inclusive practices” in private and public institutions like corporations, places of work, and universities. This may take the form of promoting forms of education for children that emphasize the values of mutual tolerance and acceptance across lines of race, culture, or ethnicity; it may take the form of universities celebrating Juneteenth or the birthday of Martin Luther King, Jr.; or it may take the form of hosting a reading of the poetry of Audre Lorde or the fiction of James Baldwin. The goal of such activities is to actively encourage the psychology of acceptance and mutual understanding across social, racial, and cultural divisions in our society.

There are obvious ways of connecting “institutional encouragement of inter-group tolerance and respect” to the agreed-upon mandate of achieving “equal rights of liberty, participation, and opportunity” for all groups. It can be argued that persistent prejudice in a segment of a population is itself a mechanism through which out-group disadvantage is maintained (Mills, The Racial Contract). Residential segregation leads to other social ills for the segregated group (health disparities, for example) and so the mechanisms that lead to the persistent residential segregation, which surely include latent prejudice and negative affect towards the out-group, can legitimately be targeted by state programs aiming at reducing prejudice and antagonism. So “anti-prejudice” and “tolerance-enhancing” programs and educational initiatives are a legitimate mechanism for achieving progress towards civil and social equality, to the extent that we can demonstrate that these efforts are efficacious. This is an instrumental justification for public and private programs aimed at reducing prejudice, stereotype, disrespect for others, and outright antagonism.

However, MLK wanted to argue for a stronger view: that achieving a harmonious, tolerant, and mutually respectful multicultural society is good in itself as an important and beneficial goal for the whole of society — not merely instrumentally valuable for achieving civil equality. In fact, it is a fundamental value for a modern democracy. Can this position be supported within liberal political philosophy? And can achieving a tolerant and respectful citizenry be considered a legitimate duty of the state?

There is a strong basis for arguing that a political philosophy articulating the values of a respectful, compassionate, and free community of equals across all groups in society is an excellent basis for thinking about the challenges of twenty-first century social life. We can offer both intrinsic and instrumental reasons for thinking that progress towards this ideal — a version of Martin Luther King Jr.’s Beloved Community — would lead to a better social world. What is less clear is whether a state organized around the principles of traditional liberalism can effectively adopt — or even tolerate — the steps needed to transform society in a more tolerant and compassionate direction. Seen in this light, the struggle over DEI raises profound questions about the possibility of progress towards real racial justice in our democracy.

(Tommie Shelby and Brandon M. Terry’s edited collection, To Shape a New World: Essays on the Political Philosophy of Martin Luther King, Jr., provides an excellent and detailed new beginning for understanding MLK’s political philosophy.)

The consequences of a trillion dollars

Many thoughtful people in the US are concerned about the effects that the 2025 federal budget reconciliation law will have on poor people in many states who are currently enrolled in Medicaid health coverage. KFF has put together a comprehensive analysis of the implications of this omnibus act for Medicaid patients here. I asked Gemini for a summary of the impact this massive change in Medicaid will have, with over $1 trillion reductions in Medicaid spending over the coming ten years(!). Here is the analysis Gemini provided:

________

QUERY:  what are the main effects of the 2025 budget bill on Medicaid recipients?

GEMINI RESPONSE 8/21/25

The 2025 budget bill, also known as the One Big Beautiful Bill Act (OBBBA), includes significant changes to the Medicaid program, which is expected to have a considerable impact on recipients. 

Here are the main effects of the 2025 budget bill on Medicaid recipients:

• Significant funding cuts: The bill implements an estimated $1.035 trillion reduction in federal Medicaid spending over ten years. These cuts will necessitate states either increasing their own funding or reducing services, potentially affecting various benefits, according to the Center on Budget and Policy Priorities.
• Work requirements: The bill introduces work reporting requirements, referred to as “community engagement” activities, for most Medicaid expansion enrollees aged 19 to 64. These recipients will need to work, volunteer, or participate in educational activities for at least 80 hours per month to maintain their eligibility.
• Increased paperwork and administrative burdens: The new requirements will place a greater administrative burden on recipients to verify their eligibility and for states to manage the process, potentially leading to some eligible individuals losing coverage due to difficulties navigating the system.
• More frequent eligibility redeterminations: States will be required to conduct eligibility redeterminations for Medicaid expansion adults every six months, rather than annually, starting January 1, 2027. This increased frequency could lead to coverage losses for those unable to complete the renewal process on time or with changes in circumstances.
• Cost-sharing requirements: States are now mandated to impose cost-sharing requirements of up to $35 per service for Medicaid expansion enrollees with incomes above 100% of the federal poverty level, says the Center for American Progress. Out-of-pocket costs will remain capped at 5% of family income.
• Restrictions on immigrant eligibility: Some categories of lawfully present immigrants will lose eligibility for Medicaid and Medicare under the bill, notes the Center for Medicare Advocacy.
• Potential impacts on specific services: Reductions in federal funding might lead states to cut optional benefits, including home- and community-based services (HCBS) that are crucial for individuals with disabilities and older adults.
• Increased risk for certain groups: Older adults, people with disabilities, and individuals in rural areas are particularly vulnerable to losing coverage or facing increased costs due to these changes. 

It’s important to remember that some provisions of the bill may have delayed effective dates, and future policy changes could still occur. 

________

So why is this a big deal for lower- and lower-middle income people in the United States? Because it seems likely that millions of people will lose affordable health coverage, and many thousands of them will sicken and die from preventable illnesses. The Center on Budget and Policy Priorities (link) estimates that as many as 36 million people are at risk of losing Medicaid eligibility as a result of new work requirements. The Modern Medicaid Alliance is collecting stories of individual people whose Medicaid benefits have been (or will soon be) cancelled (link). These are short items drawn from various news outlets, and they paint a tragic picture from all over the country of the situations that will confront struggling low-income people, young and old, when these new Medicaid reductions begin to hit.

Let’s recall as well that health disparities in the US are severe by almost any measure. Here are disparities of infant mortality in NE Florida noted by Northeast Florida Healthy Start (link).

Health Disparities by Race

And here is a comparison of urban and rural mortality statistics offered by the Rural Health Information Hub (link).

Does anyone seriously doubt that reductions in Medicaid eligibility to millions of individuals will worsen these disparities? And for what? To reduce taxes on the top 1% of income earners in the US.

Caillebotte's silences

A current exhibition of the paintings of Gustave Caillebotte (1848-1894) at the Art Institute in Chicago is quite remarkable. It demonstrates the eye, the hand, and the sensibility of this great late-Impressionist painter. But the exhibition is remarkable in another way as well: there is almost no evidence in the paintings on exhibit, or the curatorial texts that support the exhibition, that conveys the intense and prolonged social, political, and military conflict of the period from the late 1840s through the defeat of the Paris Commune (1871).

Caillebotte himself served in the French military during the siege of Paris by the Prussian Army (1870-1871). In the text describing the exhibition at the Musée d’Orsay in Paris a curator writes briefly of his service: “But during the Franco-Prussian War (1870-1871), [Caillebotte] was drafted into the 7th Battalion of the Garde Nationale Mobile de la Seine and assigned to the defense of Paris” (link). The detail about his service in the 7th Battalion of the Garde Nationale Mobile is especially telling. Michael Howard describes the Battle of Buzenval in these terms:

The battle of Buzenval, as it was to be called, settled the fate of Paris. More, it destroyed once for all the belief that a People in Arms could overwhelm a trained enemy by sheer numbers and burning zeal. It was the action for which the clubs had for so long yearned—the sortie en masse. Nearly 90,000 men were involved, of whom about half came from the Garde Nationale, and at dawn on 19th January they debouched from Mont Valérien, and advanced against the German defences between Bougival and St Cloud along a four-mile front. (Howard, The Franco-Prussian War, p. 373).

This battle was yet another disaster for the French military. It essentially sealed the fate of the besieged city and forced surrender of the last part of France still resisting German control. If Caillebotte was an active participant in this battle, he would have had traumatic and transformative experiences of war; and if he was held in reserve in the city during this final break-out attempt, he would have had personal knowledge of the significance and suffering created by the siege of Paris and the several unsuccessful efforts to break the siege through counter-attacks. Surely this is an important element in his development as an observant human being and a painter. And yet there is no evidence in his paintings of the impression the trauma of Paris may have had upon him. It would seem that this is an important contextual feature that should play a highlighted role in the curatorial presentation of the exhibition; but it does not.

Stéphane Guégan, a scientific advisor to the Musée d'Orsay in Paris and author of Caillebotte: Peintre des extrêmes, considers the “military presences” in Caillebotte’s paintings. Referring to the first exhibition of the Impressionist group in 1874, he notes that “the [first Impressionist] exhibition did contain a few resounding echoes of the catastrophic situation from which the country was barely emerging: the crushing defeat of the Franco-Prussian War and the ensuing upheaval of the Paris Commune” (“The Shared War”, Gustave Caillebotte: Painting Men, p. 42). But virtually no such references occur in the exhibition that has travelled from Paris to Los Angeles to Chicago. Guégan notes several exceptions: the painting of a soldier in uniform and an almost imperceptible representation of a uniformed soldier in Le Pont de l’Europe.

But neither image gives any sense of the true military catastrophe of the Franco-Prussian War or its aftermath. Instead, the curators have chosen to organize their ideas about Caillebotte’s paintings around the form of masculinity represented by his work.

Contrast Caillebotte’s silence in his painting with that of Jean-Louis Ernest Meissonier, a generation senior to Caillebotte. Here is Meissonier’s 1849 depiction of a massacre he apparently witnessed following the defeat of the workers’ uprising in June 1848.

And here is Meissonier’s 1884 painting representing a scene of death and destruction during the siege of Paris in 1870:

Finally, here is a daguerrotype of a poignant scene following the massacre of Communards following the fall of the Commune in 1871:

The traumas represented in these images were part of the experience and memory of the people of Paris during those decades, including both Meissonier (who was 33 at the time of the 1848 workers’ uprising) and Caillebotte (who was born in 1848 and was a serving member of the Garde Nationale Mobile in 1870-1871 during the final months of the siege of Paris and the suppression of the Commune itself (by French forces). How could either of these painters not have been deeply affected by these traumatic events of contemporary French history? Meissonier’s paintings take note of these fundamental facts, but Caillebotte’s do not. And yet the exhibition gives no historical context at all that would highlight these important and surely formative events.

And what about Meissonier? His depiction of the massacre of workers in 1848 might suggest that he was offering sympathy and homage to the working class men and women who rose up in June, 1848. Alexis de Tocqueville and Alexander Herzen, both observers of the fighting in Paris in June 1848, offered sympathy and sorrow for the violence that overwhelmed the workers’ uprising. Here are comments offered by Herzen:

I listened to the thunder and the tocsin and gazed avidly at this panorama of Paris; it was as though I was taking my leave of it. At that moment I loved Paris passionately. It was my last tribute to the great town; after the June days it grew hateful to me. On the other side of the river barricades were being raised in all the streets and alleys. I can still see the gloomy faces of the men dragging stones; women and children were helping them. A young student from the Polytechnic climbed up on to an apparently completed barricade, planted the banner and started singing the Marseillaise in a soft, sad, solemn voice; all the workers joined in and the chorus of this great song, resounding from behind the stones of the barricades, gripped one’s soul. . . . The tocsin was still tolling. Meanwhile, the artillery clattered across the bridge and General Bedeau standing there raised his field-glasses to inspect the enemy positions. . . . (From the Other Shore, After the Storm, 46)

But this is not the current interpretation of Meissonier’s work. Rather, critics have suggested that the 1849 painting of the massacre at the barricade conveys a middle-class view of the insurrection, and serves as a caution for the future: “insurrection leads to massacre and death”, while the 1888 painting conveys a sense of patriotism and heroism.

An element of historical change that is entirely evident in Caillebotte’s paintings is the transformation of Paris by Baron Haussmann at the direction of Napoleon III. The “Paris Street: Rainy Day” painting above reflects the Haussmannization of Paris — the broad avenues, the “modernization” of life in the city, and the destruction of working class residential areas. This is a central theme in T.J. Clark’s interpretation of mid-century depictions of Paris in The Painting of Modern Life:

It seems that only when the city has been systematically occupied by the bourgeoisie, and made quite ruthlessly to represent that class’s rule, can it be taken by painters to be an appropriate and purely visual subject for their art….. For the House knew well that Haussmann’s modernity had been built by evicting the working class of Paris from the centre of the city, and putting it down on the hill of Belleville or the plains of La Villette, where the moon was still most often the only street light available. And what did painters do except join in the cynical laughter and propagate the myth of modernity? (The Painting of Modern Life, p. 51)

According to Clark, there was an overriding theme of class conflict and a fear of insurrection that drove both Napoleon III and Baron Haussmann in this urban project. They were concerned to “modernize” Paris in a way that would make working class rebellion (and the barricades through which previous uprisings had proceeded) impossible; troops and cannon would be able easily to clear the avenues of insurrection.

There was no disputing that part of Haussmann’s modernity was his wish to put an end to insurrection. He stated as much himself: it was a good argument to lean on when pleading for funds from the Conseil Municipal. Years after the event, he was still musing in his Mémoires over the hidden benefits of the Boulevard Sébastopol: “It meant the disembowelling of the old Paris, the quartier of uprisings and barricades, by a wide central street piercing through and through this almost impossible maze, and provided with communicating side streets, whose continuation would be bound to complete the work thus begun. The subsequent completion of the Rue de Turbigo made the Rue Transnonain [symbolic capital of the barricades] disappear from the map of Paris!” Nor was this merely a matter of hindsight on Haussmann’s part. The details of counterrevolution weighed heavily on the planners’ minds at the time: Napoleon intervened directly in 1857 to prevent the encirclement of the Faubourg Saint-Antoine from being spoiled by a mere architect’s whim: “the construction of arcades on the Boulevard Mazas,” he wrote, “would seriously damage the strategic system of Paris.” The arcades were quietly dropped from the designs. (75-76)

Here again the current exhibition’s curators have seemingly ignored the social and political context of the Haussmannization of Paris. They emphasize the “new modernity”, the dress of the mostly bourgeois men and women passing across the boulevards, and the relaxed scenes of conversation and amusement among Caillebotte’s male friends. But there is no curatorial mention at all of the political fears and imperatives that appear to have driven Napoleon III and Baron Haussmann in this wholesale restructuring of the urban environment of Paris. And there is no suggestion in the paintings on display in the current Caillebotte exhibition of a sensibility on the part of the painter to the underlying conflicts between working class Parisians and the bourgeoisie. There are notes of awareness and sympathy for working men and women in his corpus — for example, in the painting “The Floor Scrapers”. But there is no suggestion of his own awareness of the concrete circumstances of injustice, exploitation, or unnecessary misery in his paintings.

So the social conditions of class and war seem to be almost entirely absent in Caillebotte’s work. There is no sense of “social critique” or self-awareness of upper-middle-class position in these paintings. Caillebotte is silent about the most momentous events that transformed France and Paris during his adulthood, and this is surprising. But it is also surprising that the curators have not taken more notice of this absence.

Real multicultural democracies

Chicago is a highly diverse city, and it is a good example of life in a multicultural democracy. The image above is a photo of the crowd on Navy Pier on a recent Saturday summer evening. According to local estimates, as many as 120,000 people visit Navy Pier on a Saturday night, and it is a good practical example of the benefits of multicultural democracy. The crowd is highly diverse, with adults and children from all racial groups and many ethnicities and language groups. And there is a substantial degree of social class mixing as well, from young professionals from the North side to working class families from the South and West sides of the city. Turn your head in different directions and you will hear a dozen different languages. The atmosphere is comfortable, fun, accepting, and interactive, with a Latino music performance going on in the open-air music venue, families enjoying a meal from the food court, and a beautiful view of the Chicago waterfront and skyline. It's a fun outing for all the residents of the city. (Chicago's population is about 2.8 million, so a typical Saturday night on Navy Pier in the summer draws almost 5% of the city's residents.)

What are the facts of Chicago's diversity? Chicago's population is now about 2.75 million, of whom 21% are foreign born. According to the US Census Bureau 2020 Decennial Census (link), the largest racial/ethnic communities in the city include White (36%), Hispanic/Latino (30%), Black/African-American (29%), and Asian (7%). 11% of respondents reported "two or more races" in the Census questionnaire. (It will be noted that these population groups add up to more than 100%. The Census Bureau provided some information about changes in methodology in 2020 which may account for this discrepancy; link.)

Segregation in Chicago region

So Chicago is highly diverse. However, the city remains significantly segregated by neighborhood, and these patterns of segregation produce a continuing legacy of disadvantage in terms of important measures of social wellbeing (health, economic opportunity, educational outcomes). The Metropolitan Planning Council and the Urban League have studied these trends carefully, and their "Shared Future" report (link, link) serves both to detail the facts of segregation in Chicago today and to outline some strategies for reversing these trends.

So Chicago's problems of achieving racial equality persist. And yet on a warm August evening in the center of the Chicago Loop, it is possible to see how this city is creating a climate of mutual respect and civic equality. Multiple community-based organizations do the work of striving for racial justice and establishing an inclusive community for all Chicagoans through ongoing efforts, programs, and community alliances. The city's political leadership recognizes that "unity within diversity" must be the beginnings of Chicago's public values and urban politics. And academic institutions like the University of Illinois Chicago's Institute for Research on Race and Public Policy (link) have a continuing commitment to documenting the facts about racial and ethnic equality in Chicago, and identifying policy initiatives that can lead to meaningful progress. It is possible for our society to become more just and more harmonious through our own patient collective efforts.

If we look carefully at the photo of the Navy Pier crowd, we will see something surprising looming over the horizon of this vibrant mass of multicultural humanity. We see in the distance the luxury hotel and tower developed by the president of the United States, located a half-mile up the Chicago River. The contrast could not be more striking, between the glittering symbol of the political movement that is demonizing diversity in our country, versus the social bonds and community spirit of mutual acceptance that constitute the reality of our multicultural democracy.

Langston Hughes caught much of the paradox of race in America when he wrote these lines in 1935:

Let America be America again.
Let it be the dream it used to be.
Let it be the pioneer on the plain
Seeking a home where he himself is free.

(America never was America to me.)

Let America be the dream the dreamers dreamed—
Let it be that great strong land of love
Where never kings connive nor tyrants scheme
That any man be crushed by one above.

(It never was America to me.)

And Walt Whitman was right too when he wrote, "I hear America singing, the varied carols I hear" as his own celebration of the breadth of experience of American society. America sings on Navy Pier and the many other places where citizens do better than politicians at facing the challenge of creating durable multicultural democracy. Being there reinforces one's confidence that the community and diversity of our country will prove stronger than the forces of xenophobia, mistrust, and antagonism that are being mobilized against us.

Ethnography of high-energy physics

Science proceeds through research communities whose participants share important and often distinctive features of thought and method. This is one of the key insights of the “historical turn” in the philosophy of science initiated in the 1970s (link, link), and it underlies much work within the interdisciplinary field of Science and Technology Studies. But what more specifically goes into the “denkkollectiv” (Ludwik Fleck), “research programme” (Imre Lakatos), or “disciplinary matrix” (Thomas Kuhn) of a specific scientific field? One way of gaining knowledge about those features of thinking and experimenting in specific research communities is through immersive study by ethnographers and micro-sociologists. Paul Rabinow offered an especially fruitful example of this kind of investigation in Making PCR (link). Rabinow was specifically interested in discovering the mental and material worlds of biotechnology researchers.

This book focuses on the emergence of biotechnology, circa 1980, as a distinctive configuration of scientific, technical, cultural, social, economic, political, and legal elements, each of which had its own separate trajectory over the preceding decades. It examines the “style of life” or form of “life regulation” fashioned by the young scientists who chose to work in this new industry rather than pursue promising careers in the university world…. In sum, it shows how a contingently assembled practice emerged, composed of distinctive subjects, the site in which they worked, and the object they invented. (Making PCR, 2)

And what about the most esoteric of contemporary scientific research, high-energy particle physics (link)? How does this extended network of researchers think and work as this community seeks out further features of fundamental physics? Peter Galison’s Image and Logic: A Material Culture of Microphysics is a brilliant, clear, and extensive exposition of the interface between theory and experiment in physics. Galison thinks of contemporary physics as an overlapping set of three kinds of activity: experimentation, instrumentation, and theorizing. In this book he looks at instrumentation and the machines of physical investigation as a realm that requires its own careful study — from a historical-sociological point of view as well as from an epistemic one.

These machines have a past. To walk through the laboratories of the twentieth century is to peruse an expanse of history in which physics has played many parts. Over here, film for atomic physics, X-ray film out of boxes destined for medicine; over there, a converted television camera rewired as part of a spark chamber. In this corner a piece of preparatory apparatus for a hydrogen bomb, in that a cannibalized bit of computer. Around you in the 1950s the structure of mutable, industrial-style laboratories introduced to physics in the wartime scramble to ready nuclear weapons and radar. Shaped by the exigencies of industry and war, but also shaping the practices of both, the machines of physics are part of a wider technological material culture—neither below it, nor above it. (xviii)

And Galison emphasizes that the realm of “the practice of physics” encompasses many forms of activity: institutions, social networks, extended working groups, peer-reviewed journals, and specialized forms of knowledge developed in industrial, military, and corporate spaces.

Even this penciled sketch is but a partial presentation of the multitude of worlds within physics; there were other worlds beyond. Left out are the different university and national groups participating in large experiments, not to speak of the theorists, phenomenologists, administrators, and industrialists; there are computer programmers simulating runs and figuring out how to acquire, store, and sort the data; there are postdocs running shifts. Somehow, out of it all, comes an argument. This picture of science fits badly into the narrowly construed rationality of the algorithmic, and equally badly into the image of an unreasoned struggle by opposing forces to divvy up the territory of knowledge. Physics as a whole is always in a state of incomplete coordination between extraordinarily diverse pieces of its culture: work, machines, evidence, and argument. That these messy pieces come together as much as they do reveals the presence, not of a constricted calculus of rationality, but of an expanded sense of reason. (xxii)

Moreover, Galison suggests that laboratory machines have “meaning”, in a fairly specific sense: they have been designed and adapted by intentional agents with specific explanatory goals in mind.

I will argue that laboratory machines can command our attention if they are understood as dense with meaning, not only laden with their direct functions, but also embodying strategies of demonstration, work relationships in the laboratory, and material and symbolic connections to the outside cultures in which these machines have roots. (2)

This point amounts to a denial of technological determinism — the idea that technologies (machines) have a specific and inherent logic of development. Against this view, Galison puts forward an “agentic” view of the group processes of instrumentation and experimentation. Individuals and teams make informed guesses about what kinds of probes and instruments will illuminate particular problems, and they design instruments to carry out those investigations. And we can also look at this as a “social embeddedness” conception of the physics laboratory: the physicist (theorist, experimenter, instrument designer) brings with him or her assumptions and mental frameworks drawn from the broader society in which they emerge.

Another important insight Galison offers has to do with the “logic of experimentation” itself. In the empiricist tradition there is the idea that experiments are the means through which observation enforces the constraints of evidence on theory. But Galison emphasizes throughout the book that the nature of “experiment” and “experimenter” has changed dramatically over the past two centuries — perhaps most radically in the past fifty years. “Big science” at CERN or the Fermi Laboratory necessarily involves the extended and collaborative work of thousands of experts and technicians; so who is the experimenter there? Rather, it is necessary to interpret and reinterpret the results of the data collected after high-energy collisions, and these data do not speak univocally for themselves. “It is amid these intimate bits of machines, data, and interpretations that the categories of experiment and experimenter are embodied: defined, dismantled, and reassembled” (7).

Galison offers a novel approach to the problem of “scientific incommensurability”. Introduced by Thomas Kuhn as “incommensurable paradigms” guiding related research communities, the idea has proven elusive. Galison approaches the problem from the point of view of small differences in language and vocabulary across closely related laboratory communities; he uses the anthropologist’s ideas of creoles and pidgins to capture the differences in meaning that he identifies (48). He writes:

Because the picture of physics sketched here is one of distinct but coordinated subcultures, the notion of an interlanguage is a useful decentered metaphor. In different forms the same kind of question arises; How should we think about the relation of theorists to theorists, of theorists to experimenters, of physicists to engineers, of chemists to physicists, of image instrument makers to logic instrument makers, and of the myriad of detector subgroups within a hybrid experiment one to the other? To homogenize these various groups artificially is to miss their distinct ways of going about their craft; to represent them as participating in isolated conceptual schemes “translating” back and forth is to shut our eyes to the productive, awkward, local coordination by which communities, machines, and knowledge get built. Consider three aspects of the interlanguage. (49)

Through these “interlanguages”, Galison suggests, the separate subcultures are able to communicate about the terms and procedures of their collaborations. And this suggests a practical response to W.V.O. Quine’s hypothetical worries about the “indeterminacy of translation” that he believes confronts all inter-linguistic encounters. This is an interesting and clearly formulated framework for seeking to understand the micro-level transactions across research communities in a large research project like the activities conducted at CERN or the Fermi Laboratory. Galison writes:

In many different ways this book is a working out of the following observation: pieces of devices, fragments of theories, and bits of language connect disparate groups of practitioners even when these practitioners disagree about their global significance. Experimenters like to call their extractive moves “cannibalizing” a device. (54)

There is a further point to emphasize in Galison’s approach: his consistent avoidance of the idea that “the experimental method” exists as a general and uniform exercise in empirical science. Against this idea, he emphasizes the contingency and capacity for change that historical studies of scientific episodes display — if we are alert to the fallacy of over-generalization. For this reason he explicitly denies that the episodes he considers in this book point to a common model of “experimentation” that might be incorporated into the philosophy of science or general statements about scientific method:

The chapters of this book, like the Medieval and Renaissance histories I have cited, are grounded in the local. But I resist the designation “case study” because I do not believe that there is a set of defining precepts that can be abstracted from these or other studies to “experiment in general” (or, for that matter, “theory in general” or “instruments in general”). (62)

Rather:

My question is not how different scientific communities pass like ships in the night. It is rather how, given the extraordinary diversity of the participants in physics—cryogenic engineers, radio chemists, algebraic topologists, prototype tinkerers, computer wizards, quantum field theorists—they speak to each other at all. And the picture (to the extent one simplifies and flattens it) is one of different areas changing over time with complex border zones that sometimes vanish, coalesce, and even burgeon into quasi-autonomous regions in their own right. (63)

This is history of science at its best: attuned to the contingency and heterogeneity of various scientific research practices, sensitive to the powerful influence of context (political, ideological, economic, military) on the conduct of science, and respectful as well of the quality and rigor of scientific work when it is done well.

Anthropologist Arpita Roy took up some of these questions through an extended period of field work at the European Organization for Nuclear Research (CERN) beginning in 2007, during which she interacted intellectually and practically with dozens of physicists as they performed their scientific work. The primary result of Roy’s ethnography is her recent book, Unfinished Nature: Particle Physics at CERN. The book is most interesting when the author reports and discusses specific conversations and topics that came up with a range of specialists during her field work (theorists, experimentalists, instrumentalists, engineers, computer analysts). These conversations offer the reader a basis for reaching his or her own conclusions about the micro-culture of the CERN technical environment. Also useful is her discussion of the explosion that occurred in the accelerator tunnel in September 2008 and that interrupted work for about fourteen months. And the stated goal of the book is valuable as well:

In that vein, it is not my intention to offer an exhaustive description of a science nor a prescription for a better science but to look closely at some of the presuppositions that serve in an interesting way to connect the technical procedures of a laboratory with wider principles of intellectual classification…. By presuppositions, I mean the class of beliefs that is collectively and unconsciously held by participants and of which they are unaware but that informs every aspect of scientific thinking and activity. (5)

The book is less convincing when the author turns to reflections drawn from Marx, post-modern thinkers, and other areas of philosophy. It is unclear, for example, how Marx’s conception of the division of labor is genuinely illuminating when it comes to understanding the workings of a large laboratory complex. There is a division of labor in this institution, of course; but Marx’s delineation seems to shed little light on this fact (any more than Durkheim’s discussion might have done).

Detailed inquiries into the concrete practices and mentalities found in “big science” laboratories and research institutes are important contributions to both the sociology of science and eventually to our understanding of the epistemic standing of physics. Realist philosophers of science are confident in one of the dualities criticized by Arpita Roy — the distinction between the knower and the properties of the physical world, or the distinction between subject and object — but the cognitive and social practices involved in the scientific enterprise are deeply interesting in their own right, and ethnographic studies of the ways in which scientists and engineers go about their work are deeply interesting. Ludwik Fleck attempted such studies in the 1930s, and this tradition of investigation of “science in the making” has proven to be profoundly insightful (link, link). And emphasis on extra-scientific features of “context”, including gender, race, business interests, and national security pressures is plainly relevant to the conduct of big science — the military-industrial complex described by President Eisenhower almost 75 years ago.

Arrtificial intelligence tools for historians

Historical research may seem to be a field in which AI tools will be especially useful. Historians are often confronted with very large unstructured digital collections of documents, letters, images, treaties, legal settlements, contracts, and diplomatic exchanges that far exceed the ability of a single human researcher to sift and analyze for valuable historical insights. Can emerging tools in the AI revolution help to make systematic use of such historical collections?

Earlier applications of new quantitative methods of analysis of historical data

Several earlier periods of innovation in twentieth-century historical research suggest that historians can often borrow fruitfully from new methods and analytical tools developed in other sciences. The cliometric revolution of the 1980s (Fogel and Elton 1984; Rawski 1996; Wright 2015) brought tools of econometrics, demography, and statistics more fully into play in historical inquiry. Historians have made extensive and productive use of quantitative methods borrowed from the social sciences to investigate questions concerning the health status of various historical populations and the standard of living in different cities and regions (Crafts 1980; Lee and Feng 1999; Allen 2000; Allen, Bengtsson, and Dribe 2005). These tools usually depend upon the availability of structured databases of comparable data over time—for example, census data, birth, marriage, and death records, military records of recruits, and price data for representative goods (wheat, rice, salt). There are issues of comparability, reliability, and validity that arise in these applications of large historical datasets, but these issues are no more difficult for historians than for sociologists or political scientists. Another major area of innovation was the geospatial revolution of the 1990s (Presner and Shepard 2016; Skinner, Henderson, and Yuan 2000; Thill 2020). Efforts to place historical data and events into spatial order have been very productive in suggesting new historical patterns and causal influences not visible in purely narrative accounts. G. William Skinner’s pathbreaking work on the economic regionalization of China is an outstanding example (Skinner 1977), and Peter Bol and colleagues have collaborated in the establishment of a major historical GIS database for China (Bol 2006; Bol 2007). So it is quite understandable that some contemporary historians are interested in the potential value of emerging tools of digital humanities, semantic search, and big-data analytics in their efforts to make sense of very large archives of digitized text and image materials.

However, archival collections of interest to historians present special obstacles to digital research. They are usually unstructured, consisting of collections of heterogeneous text documents, contracts, local regulations, trial documents, imperial decrees, personal letters, and artifacts and images. Moreover, the meaning of legal, political, and religious vocabulary is sometimes unclear from a modern perspective, so translation and interpretation are problematic. The written language of the documents itself is problematic. Often handwritten, interspersed with references and asides in other languages, and often using vocabulary that has no exact modern equivalent, the task of interpreting each historical document itself is challenging for the historian and for the software system. Are there tools that allow the historian to sift, summarize, categorize, and highlight the texts, sentences, and paragraphs that are included in a large archival collection? Major new capabilities have emerged in each of these areas that have substantially enhanced the ability of historians to classify and analyze very large unstructured text databases and archives. These capabilities involve advances in machine learning, large language models, semantic search tools, and big-data analytics. Like any innovation in methods of inquiry and inference, it is crucial for researchers to carefully evaluate the epistemic reliability of the tools they utilize.

Digital humanities

In the past several decades scholars in the humanities, including comparative literature, art history, and various national literatures, have explored applications of computational tools for the analysis of digital texts that permit a breadth and depth of analysis not previously available. These research efforts are now described as digital humanities. Several handbooks and overviews on digital humanities have appeared (Schreibman, Siemens, Unsworth 2004; Schreibman, Siemens, Unsworth 2016; Eve 2022). The goals of research within the field are varied, but in almost all cases the research involves computational analysis of large databases of text, image, and video documents, with the general goal of discovering large patterns that may be undetectable through traditional tools of literary or art-history analysis. Franco Moretti’s Graphs, Maps, Trees: Abstract Models for a Literary History (2005) and Distant Reading (2013) offer excellent examples. Moretti wishes to explore “world literature”; and the field of documents included in this rubric is too large for any single critic or research team to read closely all the available works in the field. Moretti writes, “A larger literary history requires other skills: sampling; statistics; work with series, titles, concordances, incipits—and perhaps also the ‘trees’ that I discuss in this essay” (2013: 67). In place of the insights of close reading, Moretti emphasizes the value of “distant reading” and the effort to discover broad and long patterns across national literatures and centuries. This requires using analytical tools of computational social science to classify texts, identify word patterns, create spatial networks, and (perhaps) to algorithmically assign markers to topics and styles in the texts subject to analysis. Martin Paul Eve writes, “Under such a model, the idea is that computational detection of style, theme, content, named entities, geographic place names, etc. could be discerned at scale and aggregated into a broader and continuous literary history that would not suffer from the same defects as a model that required one to read everything” (Eve 2022: 130).

Efforts in the digital humanities have evident relevance to the problems presented by large text and image datasets available in many areas of historical research. One promising area of application involves using big data tools of text analysis—for example, machine learning, content extraction, and semantic search—to systematically survey and classify all the documents in a collection. The impetus and initiatives of the field of “digital or computational history” are described in Siebold and Valleriani 2022 and Graham, Milligan, Weingart, and Martin 2022. The methods currently creating a great deal of interest among historians are based on joining machine learning methods, big-data analytics, and large language models (LLMs) in order to permit analysis and codification of the semantic content of documents. To what extent can emerging computational tools designed for management and analysis of large unstructured text and image databases be adapted to assist the historian in the task of assimilating, interpreting, and analyzing very large databases of historical documents and artifacts?

Pre-processing and information extraction

An avenue of research in computer science that supports analysis of large unstructured datasets containing texts and images is the field of information extraction (Adnan and Akbar 2019). Information extraction technology consists of algorithms developed to analyze patterns in text (and images or videos) to apply labels or tags to segments of the data. These are generally “big data” tools using machine learning to identify patterns in target documents or images. Adnan and Akbar put the goal of information extraction tools in these terms: “It takes collection of documents as input and generates different representations of relevant information satisfying different criteria. IE techniques efficiently analyze the text in free form by extracting most valuable and relevant information in a structured format” (Adnan and Akbar 2019: 6). In general terms, information extraction tools are expected to provide a structured basis for answers to questions like these: What is the document about? What persons or things are mentioned? What relationships are specified within the document? What events are named? The tools are often based on natural-language models that require training on large text datasets and sometimes make use of machine learning based on neural networks (Rithani et al. 2023). “The concept is to automatically extract characteristics from massive artificial neural networks and then use these features to inform choices” (Rithani et al. 2023: 14766).

A useful tool developed within the field of information extraction that has clear relevance for historians attempting to analyze large unstructured databases is named entity recognition and classification (Goyal, Gupta, and Kumar 2018). This is a group of text-analysis algorithms designed to identify meaningful information contained in a given document —for example, “person, organization, location, date/time, quantities, numbers” (Goyal et al. 2018: 22). The named entities may be specialized to a particular content area; for example, public health historians may wish to include disease and symptom names. These tools are used as a basis for pre-processing of a set of documents. The tool creates a meta-file for each document including the named entities and classes that it contains, along with other contextual information. For example, historians interested in the role that agriculture played in large periods of time may be interested in quickly identifying a selection of documents that refer to hunger, famine, or starvation. Goyal, Gupta, and Kumar carefully review the methods currently in use to identify named entities in a body of texts, including rule-based identification of named entities and machine-learning identification, with or without supervision. They emphasize that none of these methods is error-free, and false positives and false negatives continue to arise after training. This means that some lexical items in a document are either missed as referring to a named entity, or are incorrectly associated with a given named entity. Nonetheless, a historian can certainly use named-entity recognition and classification to provide a basis for important exploration and discovery in a large unstructured text database.

Keller, Shiu, and Yan (2024) provide a recent example of a machine-learning approach to automatic text analysis based on the most common large language model technique (“bidirectional encoder representations from transformers” (BERT)). They use GUWEN-BERT, a BERT model pre-trained on classical Chinese characters. They evaluate the power and accuracy of this tool in analyzing the Veritable Records of the Qing Dynasty to identify events of social unrest. The document archive is vast, encompassing more than 1,200 volumes of records from the sixth century to the end of the Qing Dynasty. Their research task is to identify episodes of social unrest, and then to classify these episodes into three categories—peasant unrest, militia unrest, and secret-society unrest (Keller et al. 2024: 4). This process of event identification and classification then permitted the researchers to seek out correlates of unrest, including fluctuations in grain prices. A useful example applying the same technology is provided by Liu, Wang, and Bol (2023), demonstrating largescale extraction of biographical information from a large collection of local gazetteers. Machine recognition of hand-written Chinese literary characters and translation of sentences and phrases in classical Chinese have made great progress in the past twenty years; Liu, Jaeger, and Nakagawa 2004, Leung and Leung 2010. This capability represents a major step forward in the ability of Chinese and Asian-language historians to make extensive use of large databases of historical documents such as the Veritable Records archives.

RAG, GraphRAG, and vector-similarity search

An important tool that has been of interest to historians exploring digital tools is retrieval-augmented generation (RAG) as a complement to LLM text generation systems. This area of research attempts to provide a basis for joining LLM query engines to specialized databases so that responses to queries will be based on data contained in the associated database. RAG tools are sometimes celebrated as solving two persistent problems arising in the application of natural-language generative chat functions based on large language models: the lack of auditability and the generation of fictitious responses (hallucinations) by the generative chat program. Kim Martineau describes a RAG tool in these terms: “Retrieval-augmented generation (RAG) is an AI framework for improving the quality of LLM-generated responses by grounding the model on external sources of knowledge to supplement the LLM’s internal representation of information. RAG implementation in an LLM-based question-answer system has two main benefits: It ensures that the model has access to the most current, reliable facts, and that users have access to the model’s sources, ensuring that its claims can be checked for accuracy and ultimately trusted” (Martineau 2024). A RAG framework is intended to allow the introduction of real, documented data into a natural language query-and-response system, and it is designed to be auditable. RAG picks up where pre-processing tools discussed previously leave off. RAG tools permit the retriever tool to parse a given query into component questions, and then to retrieve relevant data from pre-existing databases of documents (Lewis et al. 2021; Zhao et al. 2024).

RAG tools have in turn been extended with two related innovations. Vector similarity search is a semantic search tool that represents a document as a vector of abstract terms (like those identified in the discussion above of named entity identification and classification) (Mohoney et al. 2023). This further simplifies the task of querying the database for documents that are “about” one or more entities or events. A second valuable analytical tool is GraphRAG, which permits the construction of a network graph of the links among the elements in a document collection. Introduced by research scientists at Microsoft in 2024, GraphRAG was designed to permit analysis of global features of a large unstructured data collection. (See Larson and Truitt 2024, Edge et al. 2024a, and Edge et al. 2024b for technical descriptions of GraphRAG capabilities.) GraphRAG combines the data provided by RAG tools and connects these to LLM generative response systems. GraphRAG thus integrates indexing, retrieval, and generation. The key output of GraphRAG analysis of a database of text documents is a knowledge graph showing relationships among the various documents based on the content vectors associated with each document. (Experienced historians who make use of RAG and GraphRAG tools note that scaling up from moderate to large databases is challenging and computationally demanding.)

Limitations of the tools for historians

These tools suggest research strategies for historians confronting very large digital collections of documents and images. They permit computational procedures that classify and index the materials in the data archive that permit the historian to quickly identify items that are relevant to particular research questions -- the occurrence of famine, civil strife, dynastic unrest, or the transmission of ideas. And they permit natural-language query of the target database that provides suggestive avenues of further investigation for the historian. Crucially, these tools provide the ability to "audit" the results of a query by returning to the specific documents on which a response is based. The problem of "hallucination" that is endemic to large-language models by themselves is substantially reduced by tying responses to specific items in the database. And the algorithms of vector search allow the AI agent to quickly pull together the documents and "chunks" of text that are most relevant to the query.

These applications present powerful new opportunities for historians to make extensive use of very large databases of texts, but they also pose novel questions for the philosophy of history. In particular, they require that historians and philosophers develop new standards and methods for validating the computational methods that are chosen for various research tasks presented by the availability of large text collections. This means that we need to examine the strengths and limitations of each of these methods of analysis. Crucially, the designers and researchers of these tools are quite explicit in acknowledging that the tools are subject to error: the problem of hallucination is not fully removed, the content database itself may be error-prone, there may be flaws and limitations inherent in the training database in use, and any errors created during the information-extraction stage will be carried forward into the results. It is therefore incumbent upon the historian who uses such tools to validate and evaluate the information provided by searches and natural language queries. Nothing in the design of these tools suggests that they are highly reliable; rather, they are best viewed as exploratory tools permitting the historian to look more deeply into the collection of documents than traditional methods would permit. It will be necessary for historians to think critically about the quality and limitations of the information they extract from these forms of big-data analysis of historical databases.

References

Adnan, Kiran, and Rehan Akbar, 2019. “An analytical study of information extraction from unstructured and multidimensional big data,” Journal of Big Data, 6(1): 91. doi10.1186/s40537-019-0254-8

Allen, Robert C., 2000. “Economic Structure and Agricultural Productivity in Europe, 1300–1800,” European Review of Economic History, 3: 1–25.

Allen, Robert C., Tommy Bengtsson, and Martin Dribe (eds.), 2005. Living standards in the past: New perspectives on well-being in Asia and Europe, Oxford; New York: Oxford University.

Bol, Peter, 2006. “Creating the China Historical Geographic Information System,” in History in the Digital Age Symposium, University of Nebraska-Lincoln, video lecture. Bol 2006 available online.

Bol, Peter. 2007. Creating the China Historical Geographic Information System (text and slides). (Digital History Project, University of Nebraska-Lincoln). http://digitalhistory.unl.edu/essays/bolessay.php.

Crafts, N.F.R., 1980. “National income estimates and the British standard of living debate: A reappraisal of 1801–1831,” Explorations in Economic History, 17: 176–88.

Edge, Darren, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson, 2024a, “From Local to Global: A Graph RAG Approach to Query-Focused Summarization”, manuscript at arXiv.org.

Edge, Darren, Ha Trinh, Steven Truitt, and Jonathan Larson, 2024b, “GraphRAG: New Tool for Complex Data Discovery Now on GitHub”, blog post at Microsoft Research, 2 July 2024.

Eve, Martin Paul, 2022. The digital humanities and literary studies, first edition, Oxford: Oxford University Press.

Fogel, Robert William, and G. R. Elton, 1983. Which road to the past? Two views of history, New Haven: Yale University Press.

Goyal, Archana, Vishal Gupta, and Manish Kumar, 2018. “Recent named entity recognition and classification techniques: A systematic review,” Computer Science Review, 29: 21–43. Goyal, Gupta, & Kumar 2018 available online.

Graham, Shawn, Ian Milligan, Scott Weingart, and Kimberley Martin, 2022. Exploring big historical data: The historian’s macroscope, second edition, New Jersey: World Scientific.

Kamath, Uday, Kevin Keenan, Garrett Somers, and Sarah Sorenson. 2024. Large Language Models: A Deep Dive: Bridging Theory and Practice. 1st 2024. ed.: Springer Nature Switzerland: Imprint: Springer.

Keller, Wolfgang, Carol H. Shiue, and Sen Yan, 2024. “Mining Chinese historical sources at scale: A machine learning approach to Qing state capacity,” Working Paper 32982, National Bureau of Economic Research, Cambridge, MA. Keller, Shiue, & Yan 2024 available online.

Larson, Jonathan, and Steven Truitt, 2024, “GraphRAG: Unlocking LLM Discovery on Narrative Private Data”, blog post at Microsoft Research, 13 February 2024.

Lee, James Z., and Wang Feng, 1999. One quarter of humanity : Malthusian mythology and Chinese realities, 1700–2000. Cambridge, Mass.: Harvard University Press.

Leung, K. C., and C. H. Leung, 2010. “Recognition of handwritten Chinese characters by critical region analysis,” Pattern Recognition, 43(3): 949–961.

Lewis, Patrick, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela, 2021. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” manuscript available at arXiv.org.

Liu, Zhou, Hongsu Wang, and Peter K. Bol, 2023. “Automatic biographical information extraction from local gazetteers with Bi-LSTM-CRF model and BERT,” International Journal of Digital Humanities, 4: 195–212.

Liu, Cheng-Lin, Stefan Jaeger, and Masaki Nakagawa, 2004. “Online recognition of Chinese characters: The state-of-the-art,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 26: 198–213.

Martineau, Kim. 2024. "What is retrieval-augmented generation?". IBM Research. Accessed 11/23/2024. https://research.ibm.com/blog/retrieval-augmented-generation-RAG.

Mitchell, Melanie, Alessandro B. Palmarini, and Arseny Moskvichev, 2023. “Comparing humans, GPT-4, and GPT-4V on abstraction and Reasoning tasks,” manuscript at arXiv.org.

Mohoney, Jason, Anil Pacaci, Shihabur Rahman Chowdhury, Ali Mousavi, Ihab F. Ilyas, Umar Farooq Minhas, Jeffrey Pound, and Theodoros Rekatsinas. 2023. "High-Throughput Vector Similarity Search in Knowledge Graphs." arxiv>cs>arXiv:2304.01926.

Moretti, Franco, 2005. Graphs, maps, trees: Abstract models for a literary history, London; New York: Verso.

–––, 2013. Distant reading, London; New York: Verso.

Presner, Todd, and David Shepard, 2016. “Mapping the geospatial turn,” in A new companion to digital humanities, edited by Susan Schreibman, Raymond George Siemens and John Unsworth. Wiley/Blackwell.

Rawski, Thomas G. (ed.), 1996. Economics and the historian, Berkeley: University of California Press.

Rithani, M., R Prasanna Kumar, and Srinath Doss, 2023. “A review on big data based on deep neural network approaches,” Artificial Intelligence Review, 56(12): 14765–14801

Schreibman, Susan, Raymond George Siemens, and John Unsworth (eds.), 2004. A companion to digital humanities (Blackwell Companions to Literature and Culture), Malden, MA: Blackwell Pubications.

Schreibman, Susan, Raymond George Siemens, and John Unsworth (eds.), 2016. A new companion to digital humanities, Chichester, West Sussex, UK: Wiley/Blackwell.

Siebold, Anna, and Matteo Valleriani, 2022. “Digital perspectives in history,” Histories, 2(2): 170–177.

Skinner, G. William. 1977. "Regional Urbanization in Nineteenth-Century China." In The City in Late Imperial China, edited by G. William Skinner and Hugh D. R. Baker. Stanford, CA: Stanford University Press.

Skinner, G. William, Mark Henderson, and Yuan Jianhua. 2000. "China's Fertility Transition through Regional Space: Using GIS and Census Data for a Spatial Analysis of Historical Demography." Social Science History 24 (3): 613-652.

Thill, Jean-Claude (ed.), 2020. Innovations in urban and regional systems: Contributions from GIS&T, spatial analysis and location modeling, 1st edition, Cham: Springer International Publishing.

Wang, Dongbo, Chang Liu, Zhixiao Zhao, Si Shen, Liu Liu, Bin Li, Haotian Hu, Mengcheng Wu, Litao Lin, Xue Zhao, and Xiyu Wang, 2023. “GujiBERT and GujiGPT: Construction of Intelligent Information Processing Foundation Language Models for Ancient Texts,” manuscript at arXiv.org.

Zhao, Penghao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui, 2024. "Retrieval-Augmented Generation for AI-Generated Content: A Survey," manuscript at arXiv.org.