Tool Discrimination

What exactly is the tool itself? Tools are defined by Merriam-Webster as:

a: something (such as an instrument or apparatus) used in performing an operation or necessary in the practice of a vocation or profession. a scholar's books are his tools

b: an element of a computer program (such as a graphics application) that activates and controls a particular function. a drawing tool

Here, we are referring to tools as a computer program (similar to b), that takes input data, operates on it and produces output(s). We do not consider tools to be the many individual commands or functions within a specific tool (some might call such tools a toolset, or toolbox). However, we recognize that for many audiences and end-users, the output data alone is itself a "tool" to inform practice or decision making (e.g. BRAT). However, in those instances the user might be more effectively consuming the outputs of a tool (or model, similar to definition a above) via something like the Riverscapes Viewer or our Riverscapes Data Exchange.

Interface

RC tools are deployed to users through a variety of interfaces. Most tools have just one deployment interface, some have multiple.

Tool Grade

We classify the grade of our tools along a continuum of growth from innovative research ideas, through to operational tools in development that (with a little love and patience) can be run by someone other than the developer, on through to more broadly deployable professional tools that are robust and usable by any user in very different settings. Tools that have the potential to be efficiently run with greater repetition and/or across greater spatial extents or at greater resolution may be upgraded to production-grade tools whereas those that can then be deployed to much broader audiences in the most accessible platforms (e.g. Mobile Aps, Web Aps) and with ease of sharing maybe ultimately be launched as commercial-grade tools.

The Riverscapes Consortium Technical Committee ranks tool grade status using the following criteria:

Technological Readiness Levels	Tool Status	Vetted in Peer-Reviewed Literature	Source Code Documentation	Open Source	User Documentation	Easy User Interface	Scalability
TR1-2	Concept	❌	❌	❌	❌	❌	❌
TR3	Proof of Concept	🟢	❌	🟨	❌	❌	❌
TR4	Research Grade	🟢	🟨	🟨	🟨	🟨	🟢
TR5-6	Operational Grade	🟢	🟢	🟢	🟨	🟨	🟨
TR7-8	Professional Grade	🟢	🟢	🟢	🟢	🟢	🟨
TR8-9	Production Grade	🟢	🟢	🟢	🟢	🟨	🟢
TR9	Commercial Grade	🟢	🟢	🟢	🟢	🟢	🟢

❌ criteria not fulfilled
🟨 criteria partially or completely fulfilled
🟢 criteria completely fulfilled

Tool Grades Explained

Below we elaborate definitions for each tool-grade.

Concept

Research begins with ideas, and we formalize those ideas into hypotheses and concepts. Before we make an actual "tool", we might articulate a concept, sketch out how it might work (a conceptual model or diagram), or even write some pseudo code. Concepts are typically what gets pitched or proposed for research proposals, and sometimes in agenda-setting or state-of-the-science papers.

RC does not track "tools" before they exist, but the idea of a concept-grade is helpful and explicit from NASA's original technological readiness levels.

Proof-of-Concept

The vast majority of research papers published in the riverscape sciences that use a tool the author(s) developed are using proof-of-concept-grade tools. That is to say, the results produced are scientifically defensible, the methods described are also defensible, and the investigators executed those methods with tools (e.g. source code and scripts) or workflows that they authored and developed. However, another investigator wishing to repeat the work or apply the workflow to their own dataset would be unlikely able to use the tool of the authors (even if it was Open Source) without significant refactoring or manipulation.

Research-Grade

Research-Grade tools are typically those that researchers have built to make their life easier, but the idea of the tool might be alluring to other potential users. Such tools might be associated with a methodological contribution, that other researchers and users would like to apply or replicate, but for them to actually get the tool to work on their machine, with their data might be difficult to impossible. These tools are sort of "buyer beware", in that they are not really made (yet) for other users (and in the case of open-source, the "buyers" haven't paid anything... "you get what you pay for").

Research-grade tools straddle the "research" and "development" TRLs, where some degree of validation of the outputs and its potential utility has been demonstrated. However, the tool's application or utility for users outside the developers research group is limited.

Operational-Grade

Operational-grade tools are firmly in the "development" TRL realm and are based on validated and demonstrated techniques. These tools can be run by someone other than the original developer, but may require technical familiarity with command-line interfaces or specific software environments. User documentation exists but may be incomplete or assume prior domain knowledge. Operational-grade tools typically work reliably on the intended platform but may have limited error handling and require careful preparation of input data to exact specifications.

Professional-Grade

Professional-grade tools are robust, well-documented, and designed to be used by practitioners with professional expertise in the domain, though not necessarily requiring advanced technical skills. These tools feature polished user interfaces (such as QGIS plugins, ArcGIS toolboxes, or standalone GUIs), comprehensive documentation including worked examples, and produce outputs that are consistently reliable across a variety of datasets and geographic areas. A professional-grade tool can typically be run by any motivated user following the documentation, without requiring assistance from the developers.

Production-Grade

Production-grade tools are optimized for scalable, repeatable execution — often running automatically across large numbers of watersheds or geographic areas. They are typically run in batch mode via command line or cloud computing platforms such as Cybercastor, and produce Riverscapes-compliant outputs discoverable through the Data Exchange. Production-grade tools have comprehensive automated testing and quality control, and are maintained to a standard that enables consistent and reliable operation at regional or national scales. The Riverscapes network models (RSContext, VBET, BRAT, etc.) are the primary examples of production-grade tools.

Commercial-Grade

Commercial-grade tools have achieved the highest level of accessibility and are deployed to the broadest possible audiences, including non-technical end users. They are typically delivered as web applications, mobile apps, or plugins to widely-used commercial software. Commercial-grade tools maintain the highest standards for documentation, user support, error handling, and user experience design. The Riverscapes Consortium's web-based products, including the Data Exchange and PBR Explorer, are examples of commercial-grade tools.

Technological Readiness Levels

These tool-grade ideas are based on the concept of Technological Readiness Level (TRLs), as originally developed by NASA. The TRLs provide a way to discriminate between concepts and products that are in research phases, in development phases, or ready for deployment to broader audiences or market. TRLs are illustrated below (from TWI-global and formally defined by the European Union:

TRL