WIP: fix: reviews may be ahead of timeline #138

Draft
andriinikitin wants to merge 2 commits from fix-multiple-reviews-from-the-same-user into main
4 changed files with 15 additions and 17 deletions

View File

@@ -17,15 +17,16 @@ type PRReviews struct {
}
func FetchGiteaReviews(rf GiteaReviewTimelineFetcher, org, repo string, no int64) (*PRReviews, error) {
rawReviews, err := rf.GetPullRequestReviews(org, repo, no)
if err != nil {
return nil, err
}
timeline, err := rf.GetTimeline(org, repo, no)
if err != nil {
return nil, err
}
rawReviews, err := rf.GetPullRequestReviews(org, repo, no)
if err != nil {
return nil, err
}
Review

I think this is correct. The timeline could be behind rawReviews, but those rawReviews that are not on the timeline should be ignored.

If you fetch timeline after reviews, you could end up with reviews on the timeline that do not exist in the fetched reviews.

I think this is correct. The timeline could be behind rawReviews, but those rawReviews that are *not* on the timeline should be ignored. If you fetch timeline after reviews, you could end up with reviews on the timeline that do not exist in the fetched reviews.
Review

It is two generals problem. With current state the bot will re-request new review(s) if all reviews have been properly submitted and timeline is missing one or some. The reorder fixes that problem in the tests

It is two generals problem. With current state the bot will re-request new review(s) if all reviews have been properly submitted and timeline is missing one or some. The reorder fixes that problem in the tests
Review

Basically, you should end up in same situation. A race condition, ie. you add a review trigger that is not seen in timeline, means that the timeline is modified asynchronously after requests are added and you still have a race condition.

Or am I missing something in this analysis?

If the bot is requesting data here, it should still see that it requested the review already. So, that would be the bug, if it doesn't see review request.

Basically, you *should* end up in same situation. A race condition, ie. you add a review trigger that is *not* seen in timeline, means that the timeline is modified asynchronously after requests are added and you still have a race condition. Or am I missing something in this analysis? If the bot is requesting data here, it should still see that it requested the review already. So, that would be the bug, if it doesn't see review request.
Review

Yes it is the bug. It it looks that the bot checks the reviews for question 'Are there any pending reviews?' and sees answer "No". Then it checks the timeline and sees that some reviews are missing. It makes conclusion that it needs to request more.

Yes it is the bug. It it looks that the bot checks the reviews for question 'Are there any pending reviews?' and sees answer "No". Then it checks the timeline and sees that some reviews are missing. It makes conclusion that it needs to request more.
Review

ok, I think I understand where the problem is. Let me make a unit test and fix and you can merge that into this pr?

ok, I *think* I understand where the problem is. Let me make a unit test and fix and you can merge that into this pr?
Review

I've pushed my changes along with the test changes from this PR to #142 . Can you please take a look and check that this "race condition" is fixed?

In reality, the problem was that issue timeline was not consistently parsed. The initial version of the Reviews code handled only reviews. Then timeline events were tacked on, but not handled correctly. So you could have scenarios where a requested review could not be seen or in another combination, it could be seen the other way. For example, a pending review that is then revoked but was still seen as pending.

I've pushed my changes along with the test changes from this PR to #142 . Can you please take a look and check that this "race condition" is fixed? In reality, the problem was that issue timeline was not consistently parsed. The initial version of the Reviews code handled only reviews. Then timeline events were tacked on, but not handled correctly. So you could have scenarios where a requested review could not be seen or in another combination, it could be seen the other way. For example, a pending review that is then revoked but was still seen as pending.
Review

Can you please take a look and check that this "race condition" is fixed?

I didn't try to understand the changes, but I tested it quite thoroughly in #149 and I confirm that the problem is fixed as it was appearing quite often.

> Can you please take a look and check that this "race condition" is fixed? I didn't try to understand the changes, but I tested it quite thoroughly in #149 and I confirm that the problem is fixed as it was appearing quite often.
reviews := make([]*models.PullReview, 0, 10)
needNewReviews := []string{}
@@ -187,8 +188,6 @@ func (r *PRReviews) HasPendingReviewBy(reviewer string) bool {
switch r.State {
case ReviewStateRequestReview, ReviewStatePending:
return true
default:
return false
}
}
}
@@ -201,18 +200,20 @@ func (r *PRReviews) IsReviewedBy(reviewer string) bool {
return false
}
for _, r := range r.Reviews {
if r.User.UserName == reviewer && !r.Stale {
switch r.State {
res := false
for _, i := range r.Reviews {
if i.User.UserName == reviewer && !i.Stale {
switch i.State {
case ReviewStateApproved, ReviewStateRequestChanges:
return true
default:
res = true
case ReviewStateRequestReview, ReviewStatePending:
return false
}
}
}
return false
return res
}
func (r *PRReviews) IsReviewedByOneOf(reviewers ...string) bool {

View File

@@ -65,11 +65,11 @@ The testing will be conducted in a dedicated test environment that mimics the pr
| **TC-REVIEW-001** | P | **Add mandatory reviewers** | 1. Create a new PackageGit PR. | 1. All mandatory reviewers are added to both the PackageGit and ProjectGit PRs. | High |
| **TC-REVIEW-002** | - | **Add advisory reviewers** | 1. Create a new PackageGit PR with advisory reviewers defined in the configuration. | 1. Advisory reviewers are added to the PR, but their approval is not required for merging. | Medium |
| **TC-REVIEW-003** | - | **Re-add reviewers** | 1. Push a new commit to a PackageGit PR after it has been approved. | 1. The original reviewers are re-added to the PR. | Medium |
| **TC-REVIEW-004** | x | **Package PR created by a maintainer** | 1. Create a PackageGit PR from the account of a package maintainer. | 1. No review is requested from other package maintainers. | High |
| **TC-REVIEW-004** | P | **Package PR created by a maintainer** | 1. Create a PackageGit PR from the account of a package maintainer. | 1. No review is requested from other package maintainers. | High |
| **TC-REVIEW-005** | P | **Package PR created by an external user (approve)** | 1. Create a PackageGit PR from the account of a user who is not a package maintainer.<br>2. One of the package maintainers approves the PR. | 1. All package maintainers are added as reviewers.<br>2. Once one maintainer approves the PR, the other maintainers are removed as reviewers. | High |
| **TC-REVIEW-006** | P | **Package PR created by an external user (reject)** | 1. Create a PackageGit PR from the account of a user who is not a package maintainer.<br>2. One of the package maintainers rejects the PR. | 1. All package maintainers are added as reviewers.<br>2. Once one maintainer rejects the PR, the other maintainers are removed as reviewers. | High |
| **TC-REVIEW-007** | P | **Package PR created by a maintainer with ReviewRequired=true** | 1. Set `ReviewRequired = true` in `workflow.config`.<br>2. Create a PackageGit PR from the account of a package maintainer. | 1. A review is requested from other package maintainers if available. | High |
| **TC-MERGE-001** | x | **Automatic Merge** | 1. Create a PackageGit PR.<br>2. Ensure all mandatory reviews are completed on both project and package PRs. | 1. The PR is automatically merged. | High |
| **TC-MERGE-001** | P | **Automatic Merge** | 1. Create a PackageGit PR.<br>2. Ensure all mandatory reviews are completed on both project and package PRs. | 1. The PR is automatically merged. | High |
| **TC-MERGE-002** | - | **ManualMergeOnly with Package Maintainer** | 1. Create a PackageGit PR with `ManualMergeOnly` set to `true`.<br>2. Ensure all mandatory reviews are completed on both project and package PRs.<br>3. Comment "merge ok" on the package PR from the account of a package maintainer for that package. | 1. The PR is merged. | High |
| **TC-MERGE-003** | - | **ManualMergeOnly with unauthorized user** | 1. Create a PackageGit PR with `ManualMergeOnly` set to `true`.<br>2. Ensure all mandatory reviews are completed on both project and package PRs.<br>3. Comment "merge ok" on the package PR from the account of a user who is not a maintainer for that package. | 1. The PR is not merged. | High |
| **TC-MERGE-004** | - | **ManualMergeOnly with multiple packages** | 1. Create a ProjectGit PR that references multiple PackageGit PRs with `ManualMergeOnly` set to `true`.<br>2. Ensure all mandatory reviews are completed on both project and package PRs.<br>3. Comment "merge ok" on each package PR from the account of a package maintainer. | 1. The PR is merged only after "merge ok" is commented on all associated PackageGit PRs. | High |

View File

@@ -5,7 +5,6 @@ from pathlib import Path
from tests.lib.common_test_utils import GiteaAPIClient
@pytest.mark.t001
@pytest.mark.xfail(reason="The bot sometimes re-request reviews despite having all the approvals")
def test_001_automerge(automerge_env, test_user_client):
"""
Test scenario:

View File

@@ -43,7 +43,6 @@ index 0000000..e69de29
@pytest.mark.t004
@pytest.mark.xfail(reason="the bot sometimes re-requests review from autogits_obs_staging_bot despite having the approval")
def test_004_maintainer(maintainer_env, ownerA_client):
"""
Test scenario:
@@ -152,7 +151,6 @@ index 0000000..e69de29
@pytest.mark.t005
# @pytest.mark.xfail(reason="TBD troubleshoot")
def test_005_any_maintainer_approval_sufficient(maintainer_env, ownerA_client, ownerBB_client):
"""
Test scenario: