Improving her clients’ user experience was Science Data Librarian Amy E. Hodge’s top goal when, in 2022, she and her colleagues decided to improve their web form for depositing scholarly assets into the Stanford Digital Repository (SDR).
Their solution was to integrate Globus®, a platform for high performance data transfer created at the University of Chicago and the DOE’s Argonne National Laboratory, and supported here since 2015 by Stanford Research Computing.
Bringing the pieces together
The Stanford Digital Repository (SDR), distinct from other campus storage and asset management options, provides generations-long, secure preservation and access to scholarly content of enduring value to the university.
Back in 2020, Repository staff had already been using Globus for a few years to help users access library-licensed content.
“Some licenses require every user to sign an agreement. Globus allowed us to assign a particular user privileges to access a particular folder and to revoke access when the license expired,” recalls Hodge.
“We started using Globus to collect files from our users because it didn’t require us to move physical hard drives around campus, set up AWS buckets, or request access to users’ Oak or Sherlock space to retrieve files. The problem was that Globus wasn’t connected to what users and staff were doing in the web app.” The process required multiple back-and-forths with users, often over days or even weeks.
They needed a more robust system that could both streamline the process for users and free up staff time for other tasks and projects. It was two years into the Covid-19 pandemic: the ability to perform remote tasks like this quickly, easily, and without staff assistance had shifted from desirable to essential — even critical.
“More people than ever were away from campus — even students — and not many of them had access to a connection as fast as they would have on-campus. Interrupted connections were causing problems with uploads, and that's something Globus can help with.”
It wasn’t long before they asked themselves if they could automate this routine process so that they would have more time to handle truly special requests. The answer was “yes” and by January 2023 they had launched a new version of their self-deposit web tool that integrated the required Globus functionality.
Welcome feedback leads to quick-cycle upgrade
In her role as product owner, Hodge created a few demo videos of their revised app for presentations and training. One of those videos made its way to the director of Globus at the University of Chicago, Rachana Ananthakrishnan, who reached out with suggestions to make it better.
With a new set of improvements inspired by Ananthakrishnan’s feedback, work on an upgrade began in late June 2023.
Research Computing’s Karl Kornel, the group’s resident Globus expert (in addition to providing system administration, lab support services, and network expertise), played a consulting role during this phase.
“It was fun working with the talented folks over in the Libraries. They have a great use case for Globus’ Transfer and API functionality, and it was nice to see everything come together in the SDR Self-Deposit tool. And for the issues that could not be resolved with Globus’ documentation and my experience, Globus support was responsive and helpful.”
Once user testing began, results were positive, affirming that the form is now intuitive and the whole process quicker and easier. Since the launch at the beginning of August — ahead of schedule — early feedback from users has been equally encouraging.
Research software engineer and computational scientist Mark Yoder, an early adopter of the self-deposit tool, is a Research Computing team member who works closely with academic researchers in the Stanford Doerr School of Sustainability.
“Integrating Globus into SDR – already a reputable platform for data archiving and sharing, is a game changer when it comes to large data sets. These changes will undoubtedly facilitate significant improvements in research reproducibility and extend the reach of Stanford research.”
The entire Globus integration project, led by Hodge (who is also the self-deposit tool’s product owner), was the accomplishment of a 15-person core team. Developers used the Agile methodology throughout, with daily scrum meetings. The effort was supported and overseen entirely by Stanford University Libraries under Associate University Librarian Tom Cramer. (See below for a full list of project contributors.)
Find out more about what’s new with the Stanford Digital Repository at sdr.stanford.edu.
More about Globus at Stanford
Stanford’s campus-wise Globus presence began formally in 2017 when the Research Computing team set up Globus end-points on central storage platforms and began supporting Globus Connect Server installations for other groups. Research Computing, Stanford University Libraries, and the Genetics Bioinformatics Service Center co-sponsor an institution-wide subscription that unlocks features for moving and managing research data at scale.
SDR Globus Integration Project Credits
Core Team
Technical:
- Aaron Collier
- Justin Coyne
- Naomi Dushay
- Mike Giarlo
- Justin Littman
- Peter Mangiafico
- Jonathan Martin
- Ed Summers
- Vivian Wong (Technical Manager)
- Laura Wrubel
Other:
- UX Designer: Astrid Usong
- Project Manager: Hannah Frost
- Metadata Analyst: Arcadia Falcone
- Repository Manager: Andrew Berger
- Product owner: Amy E. Hodge
Additional Contributors
Dev Ops:
- Julian Morley
- Rob Smith
Other software developers:
- Chris Beer
- Nick Budak
- Justin Coyne
- Gary Geisler
- Cory Lown
SUL Director in charge:
- Tom Cramer
