Challenges for Agile way of working in research
Overview
Teaching: 40 min
Exercises: 0 minQuestions
What are challenges for an Agile way of working in research and how do I solve them?
Objectives
List challenges for an Agile way of working in research
Know the solutions to challenges for an Agile way of working in research
In this episode we will discuss and learn about challenges that arise when applying Agile principles to research projects. While some of the Agile values and principles go hand in hand with the traditional way of doing academic research, others will require an investment of the team to change their way of working.
Challenges
Discussion
What are the challenges for an Agile way of working that are specific to research? These might be challenges that are more generic to doing teamwork in research, instead of just the Agile way of working.
Uncertainty Challenge
Scientific research is all about discovering the unknown. This makes the work predictably unpredictable to an enourmous extent. The development time of a certain part of the research can range from “already solved by someone else” to “impossible to solve”. Also the requirements can suddenly change by 90 degrees based on the outcomes of a different part of the research project or the availability of data.
Cultural challenge
Many scientists are traditionally used to working individually in an isolated silo. Tasks in academia are often attributed to individuals and not to teams. A PhD-project is by default set up as a one-person project. Given this individualistic culture it is often intimidating to start with Agile practices in a scientific context.
Waterfall project setup challenge
Scientific projects are often set up in long-term project timelines following a Waterfall method. It is not common to ensure early (weekly/monthly) continuous delivery of the research outcomes to the scientific community.
Large-scale projects challenge
Scientific projects can be on a very large scale. It is hard to work together in very large teams. Also, having a continuously changing design in iterative sprints does not always lead to the best large-scale functionality and/or desired research outcomes.
Team dedication challenge
Agile is often explained in the context of a team fully dedicated to a single project. In academia, team members may often be involved in numerous projects in and outside of the team.
Product Owner challenge
For a team running multiple research projects, who acts as product owner when applying the scrum framework?
Open Source software development challenge
The nature of open source projects is that they draw (and benefit from) collaborators outside of your team, workplace and even country. Starting and contributing to open source software projects therefore leads to many dependencies on collaborators outside of the team. Not completely “owning” your product is problematic in many Agile methodologies. Approaches like Scrum may not work well or at all in such a situation.
Specific expertise challenge
For many scientific problems a very specific expertise is needed. This expertise is often even specific for a single project. This makes it hard to work together as other team members need to have or acquire this specific expertise.
What is our “product” challenge
The Agile principles and related methodologies often talk about satisfying the “customer” with bringing “value” to a “product”. But what is “value” in research? And who is the customer? It is often hard or awkward to define scientific outcomes as products with a certain functionality.
Solutions
Discussion
Can you think of solutions to these challenges?
Uncertainty Challenge
Solutions:
- Use spikes: The team could create spikes: items in the backlog that provide time for research and discovery. They are either timeboxed or their definition of done is when the specified research objectives are met.
- Do the least understood work first: Start with the most uncertain tasks and increment from there.
- Deviate from the plan: The sprint plan is not set in stone, teams should ‘negotiate the scope of the Sprint Backlog’ when needed.
- Relax “Definition of Done”: To keep a fast discovery pace, exploratory work doesn’t need to meet the same strict criteria as regular tasks.
Cultural challenge
Solutions:
- Implement flexibly: You should fit the methodology to the way of working. ‘Process-rigidity is the antithesis of agility.’
- Emphasize the “why”: Focus on the “why” for the change to Agile instead of “what” to do to make the change.
Waterfall project setup challenge
Solutions:
- Clear communication is key. Inform your collaborators on your way of working and update them frequently.
- While projects that follow the Waterfall method often do not make the targets set out at the start of the project because of the uncertainty of scientific research. Working in an Agile way enables the flexibility to change plans early on before too much time is spent on so called “dead-ends”.
- The open science movement is working on a continuous and incremental publication model to enable early adoption of scientific results and/or methods. This model fits very well with an Agile way of working.
Large-scale projects challenge
Solutions
- It makes sense to do a few design sprints to set out the bigger plan before embarking on a project.
- The Architectural Runway approach revolves around more large-scale “intentional architectures”.
Team dedication challenge
Solutions:
- Implement flexibly: Again: You should fit the methodology to the way of working. Start from where your team’s activities do overlap and where teamwork seems most beneficial.
- Keeping track: Experiment and iterate on ideas to find the best method to keep track of the different projects in your team. You can start with a Kanban board and backlog for each project seperately.
Product Owner challenge
Solutions:
- Each project has its own ‘project owner’ (usually a team member) that takes the role of product owner for that project.
Open Source software development challenge
Solutions:
- You need to identify what parts of the project you have “ownership” of, and which parts you do not, and then plan development to mitigate issues. You should adapt the chosen Agile methodology to fit the project, and not the other way around. If the software development will be (mostly) carried out by a core team of developers who work intensively on that project, with only occasional contributions from others, then Scrum can be an option. For projects that differ substantially from this, a “looser” approach may work, such as Kanban.
Specific expertise challenge
Solutions:
- Not everyone has to be involved in everything! You can help each other out on a conceptual level even without knowing all the details.
- Be explicit and open about each team member’s expertise and how it fits best within the different projects.
What is our “product” challenge
Solutions:
- Discuss this in your team! Our proposition:
- Our customers: The Scientific Community at Large, specific scientific sub-communities, non-scientific user communities, etc.
- Value for our Customers: Useful, well programmed software. Output (papers, blogs etc.). Increased FAIR-ness and quality of research. Disseminated expertise on eScience Technology.
- Product: It helps to think of all outputs of scientific research as products: Posters, papers, protocols, software, data, etc. For example, in contrast to what the first value of the Agile manifesto reads, documentation is often a very important key outcome of scientific research. A scientific instrument/software/protocol can have many issues, but when it is well documented, other groups can build on it. Vice versa, even when a scientific instrument/software/protocol is functioning perfectly, if it is undocumented, it is often useless for scientific research.
Resources
- Scrum for Data Science
- Reinventing Research: Agile in the Academic Laboratory . Includes a proposition for an Agile Manifesto for academia.
Of course, methodological issues are not the only challenge in team work. Soft skills can be just as important, if not more. We discuss this next, in episode 7.
Key Points
Agile methodologies should be slightly adapted so that they work well in a research setting
Remember to fit the methodology to your way of working