Content
View differences
Updated by Oliver Günther almost 11 years ago
The following lists implementation ideas for the requirements of the child packages. As those overlap, it is not sensible to do that on the work packages itself.
# Within OpenProject
## Limitations:
The solutions apply only to settings where filesystem access exists. When the repository is e.g. on GitHub, some of the functionality has to be disabled.
## Changes to Reposman.rb for instant creation/deletion
3 Alternatives exist:
### Extend current solution
Extend the current reposman.rb script to check for projects that have repository activated and compare that against the directory in which the repositories are stored. Delete all directories that do not belong to a project’s repository.
- \#\#18478: Extend reposman.rb to delete every repository/directory that is not part of a project.
- Creation: Already exists
Drawbacks:
- No instantaneous repo creation/deletion (violates \#18834)
- Very fragile
### Rebuild within delayed job queue
Abandon independent Reposman.rb script and make repo creation/deletion a part of OpenProject. The existing queuing mechanism (DelayedJob) could be reused to queue repository related jobs.
- \#\#18478: Issue “rm -r” command
- Creation: Strip existing reposman.rb script to get the required elements
Drawbacks:
- Running this now requires all of OpenProject to be deloyed on the server running it. But can run along with the rest of delayed processes.
### Rebuild as WebService
Transform independent Reposman.rb script into fully fledged web application (Reposman WS) that is triggered by OpenProject via HTTP Requests. E.g. A user activates the repository module in a project, that creates a job that is queued, that job calls Reposman WS which creates the repository.
- Trigger WebService when job in queue executed
<!-- end list -->
- \#\#18478: Issue “rm -r” command
- Creation: Strip existing reposman.rb script to get the required elements
Drawbacks:
- Deployment pipeline needs to be adapted.
## Attachment file size
This information is already present in the DB. It only has to be fetched and presented in the projects table. (\#18393)
## Repository file size
The information is not currently available in the DB. I propose making it available there to avoid slowing down the application for the user. That would also enable us to send an e-mail when the threshold is reached (\#18479). Showing the information in the project’s table is no big effort then (\#18393)
### SVN and GIT builtin mechanims
SVN and GIT do not provide adequate mechanisms. There is “svn list -v” but that only applies to the current revision.
### OS mechanisms
Use OS mechanisms (e.g. “du”) to get the size of the directory (check for possible abstractions, e.g. what chef uses)
Updating the filesize is a problem of it’s own.
#### Cron Job
Run cron job to update information. This might become awfully slow on large installations. installations like MyProject. It could be improved by first checking whether a directory has been touched.
Drawback:
- Another cron job that has to be configured and might fail
- Time between updating the repository and executing the cron job leads to outdated information. This time cannot be reduced beyond a certain point as the “du” information will take time.
#### Event based and intelligent
Update file size information when the user changed something in the repository. For that we could trigger a delayed job whenever a user authenticates against OpenProject for an SVN access. The authentication call against OP would have to be extended to include the action the user want’s to perform on the repository. If the action changes the size of the repo (e.g. adding, modifying, deleting) a job is run to asynchronously to update the file size stored in the DB.
Needs extension of:
- repo\_auth Pearl script to send the information on what action is desired to the OP authentication endpoint
- the OP authentication endpoint to accept the action and trigger a delayed job if the file size is likely to change
Drawback:
- more complicated than the cron job solution.
- if a lot of accesses happen within a short period of time, the delayed jobs could stack up and cause unnecessary work as the first job, when it is his delayed turn, will already update the size to the correct information. The rest are just no-ops then. The impact of those could however be reduced.
#### Event based but dumb
Like “Event based and intelligent” but no changes to the Pearl script and the authentication endpoint. Instead, simply note every access to a repository and trigger an update of that repository.
Drawbacks:
- Even worse behaviour when a lot of accesses happen within a short period of time (see “Event based and intelligent”).
## Proposed solution for the changes within OP
- [Abandon reposman.rb script and rebuild within delayed job](https://community.openproject.org/#122-Rebuild-within-delayed-job-queue). The drawback of this, that the functionality of creating repositories on servers other than the ones OpenProject is running on does not affect any of our clients. Removing the reposman.rb script reduces complexity. Later on, the WS based solution could build on this one.
- a delayed job is run when the repository module is activated (instant creation as in \#18834)
- ? a delayed job is run when the repository module is deactivated ?
- probably to risky but not doing it will leave unnecessary repositories in the system, so maybe we need a manual trigger for deletion.
- a delayed job is run when a project is deleted (\#18478)
- Attachment’s file size information already exists
- [Use OS mechanisms to collect the data](https://community.openproject.org/#142-OS-mechanisms) and trigger the update “Event based but dumb” so
- a delayed job is run for every access to the repository. This has to be made intelligent enough to reduce load to a minimum by avoiding unnecessary job executions. **There is a risk of high load here**
- When updating the storage information for a project and if the updated information reaches a threshold, an E-Mail is send to admins (\#18479)
- If a repository is not controlled by OpenProject (e.g. when it is on GitHub, the above functionality does not exist and is disabled)
- The two columns for attachment and repository size are added to the project table (\#18393)
- All installations currently employing reposman.rb need to have the cron job for this removed (setting in chef).
- Plugin openproject-stripped\_repository\_settings has to be adapted/removed.
The way \#18814 and \#17639 are specified, they can be handled independently from the rest of the requirements.
# Notes on implementation ideas on SAAS side
## Terms
- **Openproject-multitenancy (OPM)** - a plugin (ruby gem) which extends Openproject with SAAS functionality.
- **OPM API** - a custom API designed for creating / deleting organizations (instances) and deleting users.
- **Customer-Care** - user interface for a marketing manager, located at admin.openproject.com.
- **Augur** - the database and backend API for Customer-Care.
- **Reeve\_cron** - the current architectural bridge between Augur and OPM API, which is started every minute.
- **Openproject core APIv3 (APIv3)** - http://opf.github.io/apiv3-doc/
## Requirements
An Openproject tenant has the information per project regarding:
- Attachments’ size (already exists);
- Repositories’ (SVN / GIT) size (has to be implemented, see above).
## Implementation approaches
### Event-driven approach
#### Additional requirements
1. APIv3 should support not session-based authentication (\#\#18558).
2. APIv3 should extend information of an organization with storage size data (in general and per project).
3. SAAS team should get used to some model of delayed jobs, like Amazon SWF (the separate task should be created).
#### Implementation
1. Modification of attachments’ size or repositories’ size trigger a delayed job.
2. The delayed job requires information about the organization from APIv3 and call Augur API to update the information.
3. Augur should implement the new API end-point and update the DB with provided information.
4. Customer-Care displays the information per organization and in detailed view per projects.
#### Estimation
15 Points
### Fall-back approach
#### Pros and cons
- pros:
- the fastest solution;
- can be implemented right now.
<!-- end list -->
- cons:
- does not use API V3;
- can stay a permanent solution not temporary.
#### Implementation
1. OPM API has a new end-point which provides storage size of organizations and their projects.
2. Reeve\_cron gets the information regarding storage size of organizations and their projects from OPM API and passes it into Augur API end-point.
3. Augur should implement the new API end-point and update the DB with provided information.
4. Customer-Care displays the information per organization and in detailed view per projects.
The implementation could be split (to speed up) in two parts:
1. organization’s level, without information about projects;
2. project’s level, with information about projects.
However this splitting could increase the total number of Points (working day of a developer)
#### Estimation
15 Points
# Within OpenProject
## Limitations:
The solutions apply only to settings where filesystem access exists. When the repository is e.g. on GitHub, some of the functionality has to be disabled.
## Changes to Reposman.rb for instant creation/deletion
3 Alternatives exist:
### Extend current solution
Extend the current reposman.rb script to check for projects that have repository activated and compare that against the directory in which the repositories are stored. Delete all directories that do not belong to a project’s repository.
- \#\#18478: Extend reposman.rb to delete every repository/directory that is not part of a project.
- Creation: Already exists
Drawbacks:
- No instantaneous repo creation/deletion (violates \#18834)
- Very fragile
### Rebuild within delayed job queue
Abandon independent Reposman.rb script and make repo creation/deletion a part of OpenProject. The existing queuing mechanism (DelayedJob) could be reused to queue repository related jobs.
- \#\#18478: Issue “rm -r” command
- Creation: Strip existing reposman.rb script to get the required elements
Drawbacks:
- Running this now requires all of OpenProject to be deloyed on the server running it. But can run along with the rest of delayed processes.
### Rebuild as WebService
Transform independent Reposman.rb script into fully fledged web application (Reposman WS) that is triggered by OpenProject via HTTP Requests. E.g. A user activates the repository module in a project, that creates a job that is queued, that job calls Reposman WS which creates the repository.
- Trigger WebService when job in queue executed
<!-- end list -->
- \#\#18478: Issue “rm -r” command
- Creation: Strip existing reposman.rb script to get the required elements
Drawbacks:
- Deployment pipeline needs to be adapted.
## Attachment file size
This information is already present in the DB. It only has to be fetched and presented in the projects table. (\#18393)
## Repository file size
The information is not currently available in the DB. I propose making it available there to avoid slowing down the application for the user. That would also enable us to send an e-mail when the threshold is reached (\#18479). Showing the information in the project’s table is no big effort then (\#18393)
### SVN and GIT builtin mechanims
SVN and GIT do not provide adequate mechanisms. There is “svn list -v” but that only applies to the current revision.
### OS mechanisms
Use OS mechanisms (e.g. “du”) to get the size of the directory (check for possible abstractions, e.g. what chef uses)
Updating the filesize is a problem of it’s own.
#### Cron Job
Run cron job to update information. This might become awfully slow on large installations.
Drawback:
- Another cron job that has to be configured and might fail
- Time between updating the repository and executing the cron job leads to outdated information. This time cannot be reduced beyond a certain point as the “du” information will take time.
#### Event based and intelligent
Update file size information when the user changed something in the repository. For that we could trigger a delayed job whenever a user authenticates against OpenProject for an SVN access. The authentication call against OP would have to be extended to include the action the user want’s to perform on the repository. If the action changes the size of the repo (e.g. adding, modifying, deleting) a job is run to asynchronously to update the file size stored in the DB.
Needs extension of:
- repo\_auth Pearl script to send the information on what action is desired to the OP authentication endpoint
- the OP authentication endpoint to accept the action and trigger a delayed job if the file size is likely to change
Drawback:
- more complicated than the cron job solution.
- if a lot of accesses happen within a short period of time, the delayed jobs could stack up and cause unnecessary work as the first job, when it is his delayed turn, will already update the size to the correct information. The rest are just no-ops then. The impact of those could however be reduced.
#### Event based but dumb
Like “Event based and intelligent” but no changes to the Pearl script and the authentication endpoint. Instead, simply note every access to a repository and trigger an update of that repository.
Drawbacks:
- Even worse behaviour when a lot of accesses happen within a short period of time (see “Event based and intelligent”).
## Proposed solution for the changes within OP
- [Abandon reposman.rb script and rebuild within delayed job](https://community.openproject.org/#122-Rebuild-within-delayed-job-queue). The drawback of this, that the functionality of creating repositories on servers other than the ones OpenProject is running on does not affect any of our clients. Removing the reposman.rb script reduces complexity. Later on, the WS based solution could build on this one.
- a delayed job is run when the repository module is activated (instant creation as in \#18834)
- ? a delayed job is run when the repository module is deactivated ?
- probably to risky but not doing it will leave unnecessary repositories in the system, so maybe we need a manual trigger for deletion.
- a delayed job is run when a project is deleted (\#18478)
- Attachment’s file size information already exists
- [Use OS mechanisms to collect the data](https://community.openproject.org/#142-OS-mechanisms) and trigger the update “Event based but dumb” so
- a delayed job is run for every access to the repository. This has to be made intelligent enough to reduce load to a minimum by avoiding unnecessary job executions. **There is a risk of high load here**
- When updating the storage information for a project and if the updated information reaches a threshold, an E-Mail is send to admins (\#18479)
- If a repository is not controlled by OpenProject (e.g. when it is on GitHub, the above functionality does not exist and is disabled)
- The two columns for attachment and repository size are added to the project table (\#18393)
- All installations currently employing reposman.rb need to have the cron job for this removed (setting in chef).
- Plugin openproject-stripped\_repository\_settings has to be adapted/removed.
The way \#18814 and \#17639 are specified, they can be handled independently from the rest of the requirements.
# Notes on implementation ideas on SAAS side
## Terms
- **Openproject-multitenancy (OPM)** - a plugin (ruby gem) which extends Openproject with SAAS functionality.
- **OPM API** - a custom API designed for creating / deleting organizations (instances) and deleting users.
- **Customer-Care** - user interface for a marketing manager, located at admin.openproject.com.
- **Augur** - the database and backend API for Customer-Care.
- **Reeve\_cron** - the current architectural bridge between Augur and OPM API, which is started every minute.
- **Openproject core APIv3 (APIv3)** - http://opf.github.io/apiv3-doc/
## Requirements
An Openproject tenant has the information per project regarding:
- Attachments’ size (already exists);
- Repositories’ (SVN / GIT) size (has to be implemented, see above).
## Implementation approaches
### Event-driven approach
#### Additional requirements
1. APIv3 should support not session-based authentication (\#\#18558).
2. APIv3 should extend information of an organization with storage size data (in general and per project).
3. SAAS team should get used to some model of delayed jobs, like Amazon SWF (the separate task should be created).
#### Implementation
1. Modification of attachments’ size or repositories’ size trigger a delayed job.
2. The delayed job requires information about the organization from APIv3 and call Augur API to update the information.
3. Augur should implement the new API end-point and update the DB with provided information.
4. Customer-Care displays the information per organization and in detailed view per projects.
#### Estimation
15 Points
### Fall-back approach
#### Pros and cons
- pros:
- the fastest solution;
- can be implemented right now.
<!-- end list -->
- cons:
- does not use API V3;
- can stay a permanent solution not temporary.
#### Implementation
1. OPM API has a new end-point which provides storage size of organizations and their projects.
2. Reeve\_cron gets the information regarding storage size of organizations and their projects from OPM API and passes it into Augur API end-point.
3. Augur should implement the new API end-point and update the DB with provided information.
4. Customer-Care displays the information per organization and in detailed view per projects.
The implementation could be split (to speed up) in two parts:
1. organization’s level, without information about projects;
2. project’s level, with information about projects.
However this splitting could increase the total number of Points (working day of a developer)
#### Estimation
15 Points