Much of economic and social life now unfolds in digital environments. Consumer journeys happen on e-commerce platforms, and labor market signals appear in online job listings. Housing supply barriers reveal themselves through local planning portals. Inequality itself is being reproduced in digital spaces – from algorithmic bias in lending decisions to disparities in the quality of search results based on geographic location.
Research published in Nature in 2025 has documented how digital inequality manifests in online social networks, with neighborhood and racial disparities affecting access to information. Other studies have found that social structural mechanisms underlie the digital divide, with significant implications for how different communities experience the internet.
Researchers have a responsibility to document these dynamics rigorously and ethically. The digital economy has grown to $4.9 trillion globally, nearly doubling since 2020, and now accounts for 18% of U.S. GDP. This shift has significant implications for how we study society. If scholars seek to understand our lives online, research tools must be capable of capturing that vast reality.
The gap between research needs and data access
Traditional research methods – surveys, interviews, manual data collection – remain valuable. But when scholars need to uncover broad patterns taking shape across thousands or millions of data points, those methods are often too slow, too narrow, or too difficult to scale.
Recognizing this challenge, policymakers have begun to act. In October 2025, new rules under the EU’s Digital Services Act came into force, allowing researchers to gain unprecedented access to data from very large online platforms to study their societal impact. As Executive Vice President for Technological Sovereignty, Security and Democracy at the European Commission, Henna Virkkunen noted at the time that creating a safer online environment starts with studying the risks, and that independent researchers now have new pathways to study platform data.
Yet even with this progress, significant barriers remain. A 2025 mapping by GLOBSEC found that APIs, delays, and denials still block many researchers from obtaining the data they need. Even with new regulatory support, independent researchers still face major obstacles to accessing various data points, including basic data shared publicly on social platforms. The asymmetry is striking: this same information often moves quietly through commercial channels, while those who work with it openly and accountably run into the most friction.
Datafication without access creates a knowledge gap
The challenge extends beyond platform data. The commentary noted that instead of a steady expansion of access, we are now witnessing what CEPS has called a “data winter”. Access to data for research has narrowed across several domains, and even where new legal pathways exist, significant barriers remain in practice. Broader political retrenchment risks weakening the very infrastructures designed to enable responsible reuse.
This has real-life impacts, directly affecting quality of life for millions. For example, in the UK, recent government data shows that planning authorities had over 106,000 applications on hand as of late 2025. However, researchers at Warwick University have noted that decades of planning targets have not solved the UK’s housing problems, and understanding the barriers to supply requires better access to planning data.
In other words, researchers having better access to the existing data can inform policy and help solve major societal issues. Conversely, investing in uninformed solutions is like building houses where no one wants to live.
What partnerships with researchers reveal
As the European Commission has made clear, researchers’ access to platform data plays an important role in understanding systemic risks in digital environments. If public factual data is available only through restrictive or expensive arrangements, smaller companies, startups, and independent researchers will be left behind.
Our experience working with academic institutions has reinforced a broader point: researchers are not simply looking for more data. They are looking for ways to build datasets that are more representative, more rigorous, and better suited to the questions they are trying to answer.
Through Project 4β, Oxylabs’ pro bono initiative supporting academic and public-interest research, we have partnered with researchers across institutions, including Northwestern University, Warwick University, Aston University, Carnegie Mellon University, and the University of Michigan.
The projects span a wide range of disciplines, including investigations into how online search results vary by geographic location. Studying how local planning regulations affect housing supply. Analyzing labor market trends and identifying bias in hiring practices. Understanding consumer behavior in e-commerce. Educating future data scientists on the legal and ethical landscape of web scraping.
What connects these projects is a shared methodological reality. The web handles much of our communication and interaction, from strangers talking to each other to brands reaching their customers. This is where huge amounts of data on how society functions and transforms are created and change rapidly. Researchers need tools that enable them to track and document these changes.
Without scalable access to public web data, some research is simply impossible, while a lack of data impairs other research. Even when manual collection and surveys are possible alternatives, they are time-consuming, less efficient, and limited in scale and speed.
These partnerships also highlight a less obvious but equally important dimension: the need for clarity on legal and ethical standards. While ultimately the job is for the policymakers and judges, a strong community of experts can make an impact by talking to each other and the public. Working together has allowed academic institutions to root discussions of appropriate action in legal precedents formed over the past two decades.
This is why at OxyCon, a technology and data conference, we give a platform for journalists and researchers as well as industry representatives. They are part of the automated web data access community just as much as we are.
A shared responsibility
Public web data is now essential to many kinds of research, which brings new responsibilities for everyone involved. Researchers must treat this data with the same rigor, transparency, and ethical standards they apply to any other source.
However, the biggest responsibility lies with those who control data access, whether through infrastructure or regulation. Privileged positions regarding the infrastructure should be used to enable better access rather than turning public data into private property.
Most importantly, ensuring equal, fair, and responsible access is a vital policy choice. One that leads to better future choices informed by better research.
