Thoughts on Future Trust
A Blog post by Wim Hugo (WDS-SC Vice-chair)
The ICSU World Data System (ICSU-WDS) and the Data Seal of Approval have recently collaborated on the alignment of their respective sets of criteria for certification as a Trusted Digital Repository, and is in process of establishing a joint certification authority—the CoreTrustSeal—to manage the certification process associated with it. This activity contributes to a significant future focus on the trust that can be placed in elements of a distributed global research infrastructure, and the increased automation of its verification. However, it is the tip of the iceberg.
The WDS Knowledge Network defines many of the components of research activity for which there is some form of trusted service or infrastructure component required: ranging from the obvious need to reliably refer to research outputs, researchers, institutions, artefacts, projects, and the like, though the more complex aspects of trusted repositories, registries, vocabulary, and ontology services, to the assigning of levels of maturity, sustainability, or quality to these.
The trust that is required for research infrastructure to function properly is somewhat different to the trust that can be placed in the content that is curated by the research infrastructure—although one has to recognize that the two aspects are interrelated and, in some instances, inseparable. Furthermore, the trust that can be placed in content should ideally also distinguish between the significance and usability of that content, and its quality. These facets are not necessarily the same, but again are conflated to some extent in discussions about fitness-for-use, quality metrics, and the like.
Let’s work though these distinctions at the hand of some examples.
The main aim of a scholarly publication is to assert a claim in respect of a novel finding, and to expose that claim to peer review for the purpose of correction, as required1. One needs to distinguish the rules (criteria for trust) associated with the process of science and the value of the content. The latter is largely judged by significance, and measured—with varying degrees of usefulness—through citation indices and impact factors.
There are arguments that this stream of self-correcting progress is broken, especially in some disciplines, and this is strongly related to the criteria for trust. Such criteria are largely stated informally and implemented with varying degrees of diligence in research institutions, and are mostly delegated to peer review to determine if the result is trustworthy. Peer review purports to determine originality (not easily automated, and essentially linked to end-user value), quality (certainly possible to automate) and validity (can be partly automated).
One could—and in my view, should—argue that processes can be verified objectively and preferably automatically, and that our aim should be to certify their veracity using measurable criteria. Such validity and quality criteria could be extended to feasibility of reproduction, access to supporting datasets, and the like. References to widely used protocols and methods, standards, samples, and research pattern—increasingly linked to persistent identifiers—also increase the verifiable level of trust in the process.
Vocabulary (name) services play an increasingly important role in research infrastructures for a variety of reasons. Firstly, vocabularies and name services are critical to the realization of the semantic web and Linked Open Data: in essence, reducing ambiguity by referring precisely to a concept, entity, relationship, and/or characteristic of either. Secondly, these services are used to enhance the experiences of users and the value of knowledge by navigating the relationships that exist among them, which is conceptually captured in the WDS Knowledge Network and is increasingly implemented, for example, in projects such as Scholix. Again, one should not confuse the acceptability of the vocabulary or service content (e.g., whether all taxonomists in the world agree that a taxon is correct), and the quality of the service provided by the infrastructure component. For the first case, there may never be agreement (especially with taxonomists!); but, for the latter, it is a relatively simple matter to determine what constitutes a well-defined, standardized vocabulary or name service, and community efforts are underway to document and define these criteria. In addition to such operational requirements, one should include the need for sustainability and continued access into the reasonable future.
In general, one can distinguish—for all of the elements of the WDS Knowledge Network—a clear separation between judgements about value (significance, originality, inclusiveness, consensus, etc.) and the quality of the process (sustainability, standards compliance, reproducibility, and similar concerns). And, extrapolating this into the future, I suspect that we need to get ready for the following:
- Significant broadening of services and infrastructure that cover all aspects of the WDS Knowledge Network, as well as a parallel rise in the need for certification of these services and infrastructure. Already, there is a perceived need for the certification of repositories of open source code and of vocabulary services, to name but two.
- Increased automation of the certification of processes that is in tune with an expected, rapid upturn in artificial intelligence and machine learning. This will be needed because I have no doubt that the scientific method will be increasingly automated within the next decade or so. We are already overwhelmed by volumes of data and numbers of publications, and science cannot scale any further as it is limited by human capacity.
On the basis of the above, and with science increasingly reliant on trust in a wider context, ICSU-WDS should start focussing on defining trust criteria beyond data repositories and services, and on how to automate its assessment: this being the only really scalable solution to a problem of rapidly growing scope.
1 There is a parallel focus on review and consolidation or synthesis based on existing knowledge.