%%{init: { 'logLevel': 'debug', 'theme': 'default'} } }%%
gitGraph
commit id: "initial commit"
branch main_build
commit id: "CI build"
checkout main
commit
checkout main_build
merge main
checkout main
branch feature_a
branch feature_b
checkout feature_a
commit
commit
checkout main
commit id: "#release 0.1" type: HIGHLIGHT
checkout main_build
merge main
checkout main
branch release
commit tag: "0.1"
checkout main
commit
checkout feature_b
commit
commit
checkout feature_a
commit
checkout main
merge feature_a
checkout main_build
merge main
checkout main
checkout feature_b
commit
checkout main
merge feature_b
checkout main_build
merge main
checkout release
merge main tag: "0.2"
Project structure
The structure of OpenPipelines
Project structure
The root of the repository contains two main folders:
src, which contains the source code for components and workflows.- (optionally) the
targetfolder
Each subfolder from src contains a Viash namespace, a logical grouping of pipeline components that perform a similar function. Within each namespace, subfolders designate individual pipeline components. For example ./src/convert/from_bdrhap_to_h5ad contains the implementation for a component from_bdrhap_to_h5ad which is grouped together with other components such as from_10xmtx_to_h5mu into a namespace convert. In a similar manner as grouping components into namespaces, pipelines are grouped together into folders. However, these are not component namespaces and as such do not interact with viash ns commands.
As will become apparent later on, Viash not only provides commands to perform operations on individual components, but also on groups of components in a namespace and all components in a project. As a rule of thumb, the basic Viash commands (like viash test) are designated for running commands on individual components, while ns commands are (viash ns test) are for namespaces. When cloning a fresh repository, there will be no target folder present. This is because the target folder will only be created after components have been build.
Versioning and branching strategy
OpenPipeline tries to use of semantic versioning to govern changes between versions. An release of openpipelines uses a version number in the format MAJOR.MINOR.PATCH. Currenly, openpipelines is still at major version 0.x.y, meaning that public-facing breaking changes are possible on MINOR releases. These breaking changes will be documented in a dedicated section of the CHANGELOG that is published with each release. A PATCH release (i.e. a release where the MAJOR and MINOR version number stay the same), is used to resolve bugs with the pipeline but should not introduce breaking changes. Keep in mind that patches might introduce behavioral changes that may look breaking but are actually rectifying changes that were inadvertently introduced previously (and were in fact also ‘breaking changes’). In this case, a bug can also be released without changing the MINOR version, in a PATH release.
Between releases, development progress is tracked on Git branches. A git branch represents a snapshot of a codebase in time, to which changes can be added (i.e. committed). Eventually, all new feature or bugfixes must be reconsiled into a single branch so that a new release can be created. This process is called merging and the process of requesting the merging of two branches is called a pull request. Openpipelines follows the convention that the target branch for all pull requests is the main branch. Thus, the main branch contains the latest changes for the code and it can be considered the development branch.
Once a pull request has been approved and merged, Github Actions CI will automatically build all components (creating the target directory) and push the result to the main_build branch. In essence, the main_build branch is a copy of the main branch, but also containing the build components. Once it is time to create a openpipelines release, the Github CI release workflow is manually triggered, the components on the main branch will be build and tested. Then, the result will be pushed to the release branch and the integration tests will be run. If all tests succeeded, a new github tag and release can be created manually from the release branch.