Skip to content

[v3.1.2] Cover letters created before migrating to v3 show up separatly in patch list #556

Open
Open
[v3.1.2] Cover letters created before migrating to v3 show up separatly in patch list#556
Labels
bugdatabaseIssues with the database or database migrations

Description

@alialnu
alialnu
opened on Aug 16, 2023

Example:
This cover letter was part of this series before the migration, but it no longer is. Moreover, its state now says None.

There are many other covers with the same issue.

EDIT:
The series above actually has a cover letter as part of it, but the cover letter is listed again in the patch list:
This one is part of the series.
This one isn't.

This isn't consistent with patchsets that were created after the migration.

The issue appeared when I migrated from v2.2.6 to v3.1.2.

Activity

  1. alialnu commented on Aug 24, 2023

    @alialnu
    alialnu
    on Aug 24, 2023
    ContributorAuthor

    Ah, right, they will be different since you have new cover letters. You probably need to add a limit to this. We don't track our own created_at/updated_at fields so we need to use date as a stand-in. This should do the trick (or at least the PostgreSQL equivalent will):

    Both queries now return the same number of rows:

    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL ORDER BY patch.id;
   id
--------
  40691
  40702
  [..]
  129709
  129712
(8138 rows)
    SELECT cover.id
FROM patchwork_cover cover
WHERE cover.date <= (
    SELECT MAX(patch.date) FROM patchwork_patch patch WHERE patch.state_id IS NULL
) ORDER BY cover.id;

   id
--------
  40691
  40702
  [..]
  129709
  129712
(8138 rows)

  2. alialnu commented on Aug 24, 2023

    @alialnu
    alialnu
    on Aug 24, 2023
    ContributorAuthor

    Whoops, I forgot about the events. How about if you add the following after the first step?

    # migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET cover_id = patch_id, category = 'cover-comment-created', patch_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    Deleting duplicate patch comments still fails:

    INSERT INTO patchwork_covercomment (msgid, date, headers, content, cover_id, submitter_id)
SELECT msgid, date, headers, content, patch_id, submitter_id
FROM patchwork_patchcomment
WHERE patchwork_patchcomment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
    INSERT 0 26

    UPDATE patchwork_event
SET cover_id = patch_id, category = 'cover-comment-created', patch_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
    UPDATE 26

    DELETE FROM patchwork_patchcomment AS patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
    ERROR:  update or delete on table "patchwork_patchcomment" violates foreign key constraint "patchwork_event_patch_comment_id_4123011e_fk_patchwork" on table "patchwork_event"
DETAIL:  Key (id)=(162805) is still referenced from table "patchwork_event".

  3. stephenfin commented on Aug 26, 2023

    @stephenfin
    stephenfin
    on Aug 26, 2023
    Member

    Deleting duplicate patch comments still fails:

    [snip]

    ERROR:  update or delete on table "patchwork_patchcomment" violates foreign key constraint "patchwork_event_patch_comment_id_4123011e_fk_patchwork" on table "patchwork_event"
DETAIL:  Key (id)=(162805) is still referenced from table "patchwork_event".

    That field should have been unset in the previous statement. Perhaps it needs to be split into two statements. Can you check if it was unset?

    SELECT cover_id, patch_id FROM patchwork_event WHERE id=162805;

  4. alialnu commented on Aug 26, 2023

    @alialnu
    alialnu
    on Aug 26, 2023
    ContributorAuthor

    That field should have been unset in the previous statement. Perhaps it needs to be split into two statements. Can you check if it was unset?

    SELECT cover_id, patch_id FROM patchwork_event WHERE id=162805;

    That id belongs to a check-created event, so I guess it shouldn't have been affected by the previous query:

    SELECT cover_id, patch_id, created_check_id, category FROM patchwork_event WHERE id=162805;
     cover_id | patch_id | created_check_id |   category
----------+----------+------------------+---------------
          |    63842 |           102713 | check-created
(1 row)

  5. alialnu commented on Aug 26, 2023

    @alialnu
    alialnu
    on Aug 26, 2023
    ContributorAuthor

    By the way,
    I noticed that new cover IDs started counting from 1 after the migration:
    https://patches.dpdk.org/api/covers/1/
    The last cover ID was 40691 before the migration:
    https://patches.dpdk.org/api/covers/40691/
    I guess it's an expected side effect of removing the old CoverLetter model?

  6. stephenfin commented on Aug 30, 2023

    @stephenfin
    stephenfin
    on Aug 30, 2023
    Member

    Apologies: 162805 refers to a row in the patchwork_patchcomment table (I see https://patches.dpdk.org/comment/162805/ resolves to https://patches.dpdk.org/project/dpdk/patch/cover.1640838702.git.songyl@ramaxel.com/#162805). We need to migrate the patch_comment_id field to cover_comment_id field, in addition to migrating patch_id to cover_id.

    Let's replace:

    # migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET cover_id = patch_id, category = 'cover-comment-created', patch_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    with:

    # migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET
    cover_id = patch_id,
    cover_comment_id = patch_comment_id,
    category = 'cover-comment-created',
    patch_id = NULL,
    patch_comment_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

  7. stephenfin commented on Aug 30, 2023

    @stephenfin
    stephenfin
    on Aug 30, 2023
    Member

    By the way, I noticed that new cover IDs started counting from 1 after the migration: patches.dpdk.org/api/covers/1 The last cover ID was 40691 before the migration: patches.dpdk.org/api/covers/40691 I guess it's an expected side effect of removing the old CoverLetter model?

    Yes, that's expected behavior. I'm going to guess PostgreSQL was restarted at some point? I know MySQL restarts the autoincrement counter at the lowest available value when it's restarted. It shouldn't matter since the ID has no significance and we've intentionally started hiding it in recent releases (at least in the web UI: the REST API and deprecated XML-RPC API still need work)

  8. alialnu commented on Aug 30, 2023

    @alialnu
    alialnu
    on Aug 30, 2023
    ContributorAuthor

    Apologies: 162805 refers to a row in the patchwork_patchcomment table (I see https://patches.dpdk.org/comment/162805/ resolves to https://patches.dpdk.org/project/dpdk/patch/cover.1640838702.git.songyl@ramaxel.com/#162805). We need to migrate the patch_comment_id field to cover_comment_id field, in addition to migrating patch_id to cover_id.

    Let's replace:

    # migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET cover_id = patch_id, category = 'cover-comment-created', patch_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    with:

    # migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET
    cover_id = patch_id,
    cover_comment_id = patch_comment_id,
    category = 'cover-comment-created',
    patch_id = NULL,
    patch_comment_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    I get a different error right now:

    UPDATE patchwork_event
SET
        cover_id = patch_id,
        cover_comment_id = patch_comment_id,
        category = 'cover-comment-created',
        patch_id = NULL,
        patch_comment_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
    ERROR:  insert or update on table "patchwork_event" violates foreign key constraint "patchwork_event_cover_comment_id_861b3ec2_fk_patchwork"
DETAIL:  Key (cover_comment_id)=(162801) is not present in table "patchwork_covercomment".

  9. alialnu commented on Aug 30, 2023

    @alialnu
    alialnu
    on Aug 30, 2023
    ContributorAuthor

    By the way, I noticed that new cover IDs started counting from 1 after the migration: patches.dpdk.org/api/covers/1 The last cover ID was 40691 before the migration: patches.dpdk.org/api/covers/40691 I guess it's an expected side effect of removing the old CoverLetter model?

    Yes, that's expected behavior. I'm going to guess PostgreSQL was restarted at some point? I know MySQL restarts the autoincrement counter at the lowest available value when it's restarted. It shouldn't matter since the ID has no significance and we've intentionally started hiding it in recent releases (at least in the web UI: the REST API and deprecated XML-RPC API still need work)

    Thanks for the confirmation. I don't recall restarting the database server. The counter was reset right after the migration.

  10. stephenfin commented on Sep 4, 2023

    @stephenfin
    stephenfin
    on Sep 4, 2023
    Member

    Lovely. So when we did the first step in the migration

    # copy any recent patch comments that should have been assigned to a cover letter instead
INSERT INTO patchwork_covercomment (msgid, date, headers, content, cover_id, submitter_id)
SELECT msgid, date, headers, content, patch_id, submitter_id
FROM patchwork_patchcomment
WHERE patchwork_patchcomment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE ISNULL(patch.state_id)
);

    we will have got a new, likely different ID for each new entry in patchwork_covercomment. As such, we need to set cover_comment_id to the ID of the row in patchwork_covercomment which corresponds to the old patchwork_patchcomment row. I'm not actually entirely sure how to do that. The best I could come up with is this horribly inefficient query.

    # migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET
    cover_id = patch_id,
    cover_comment_id = (
        SELECT patchwork_covercomment.id
        FROM patchwork_covercomment
        INNER JOIN patchwork_patchcomment
        ON patchwork_covercomment.msgid = patchwork_patchcomment.msgid
        WHERE patchwork_patchcomment.id = patchwork_event.patch_comment_id
    ),
    category = 'cover-comment-created',
    patch_id = NULL,
    patch_comment_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    I've no idea if that will work or not as I still haven't got a full reproducer locally for these follow-up issues.

  11. stephenfin commented on Sep 4, 2023

    @stephenfin
    stephenfin
    on Sep 4, 2023
    Member

    Also, just for my own sanity, I think we're now at the following migration script.

    # copy any recent patch comments that should have been assigned to a cover letter instead
INSERT INTO patchwork_covercomment (msgid, date, headers, content, cover_id, submitter_id)
SELECT msgid, date, headers, content, patch_id, submitter_id
FROM patchwork_patchcomment
WHERE patchwork_patchcomment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

# migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET
    cover_id = patch_id,
    cover_comment_id = (
        SELECT patchwork_covercomment.id
        FROM patchwork_covercomment
        INNER JOIN patchwork_patchcomment
        ON patchwork_covercomment.msgid = patchwork_patchcomment.msgid
        WHERE patchwork_patchcomment.id = patchwork_event.patch_comment_id
    ),
    category = 'cover-comment-created',
    patch_id = NULL,
    patch_comment_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

# delete the duplicate patch comments
DELETE patch_comment
FROM patchwork_patchcomment patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

# delete the duplicate patches
DELETE patch FROM patchwork_patch patch WHERE patch.state_id IS NULL;

  12. alialnu commented on Sep 4, 2023

    @alialnu
    alialnu
    on Sep 4, 2023
    ContributorAuthor

    we will have got a new, likely different ID for each new entry in patchwork_covercomment. As such, we need to set cover_comment_id to the ID of the row in patchwork_covercomment which corresponds to the old patchwork_patchcomment row. I'm not actually entirely sure how to do that. The best I could come up with is this horribly inefficient query.

    The UPDATE statement fails with:

    ERROR:  more than one row returned by a subquery used as an expression

    I'm having a difficult time constructing a query to find the problematic IDs though.

  13. stephenfin commented on Sep 5, 2023

    @stephenfin
    stephenfin
    on Sep 5, 2023
    Member

    How about if we drop the subquery and do joins?

    UPDATE patchwork_event AS event
    INNER JOIN patchwork_patchcomment AS patch_comment
    ON patch_comment.id = event.patch_comment_id
    INNER JOIN patchwork_covercomment AS cover_comment
    ON cover_comment.msgid = patch_comment.msgid
SET
    event.cover_id = event.patch_id,
    event.cover_comment_id = cover_comment.id,
    event.category = 'cover-comment-created',
    event.patch_id = NULL,
    event.patch_comment_id = NULL
WHERE event.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    Again, disclaimer that this is not beyond validating that it's valid (MySQL) syntax.

  14. alialnu commented on Sep 5, 2023

    @alialnu
    alialnu
    on Sep 5, 2023
    ContributorAuthor

    Quick update while I try to rewrite the query:

    How about if we drop the subquery and do joins?

    UPDATE patchwork_event AS event
    INNER JOIN patchwork_patchcomment AS patch_comment
    ON patch_comment.id = event.patch_comment_id
    INNER JOIN patchwork_covercomment AS cover_comment
    ON cover_comment.msgid = patch_comment.msgid
SET
    event.cover_id = event.patch_id,
    event.cover_comment_id = cover_comment.id,
    event.category = 'cover-comment-created',
    event.patch_id = NULL,
    event.patch_comment_id = NULL
WHERE event.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

    Doesn't seem like it's valid syntax:
    https://www.postgresql.org/docs/current/sql-update.html

    ERROR:  syntax error at or near "INNER"
LINE 2:     INNER JOIN patchwork_patchcomment AS patch_comment

  15. stephenfin commented on Sep 5, 2023

    @stephenfin
    stephenfin
    on Sep 5, 2023
    Member

    Guess that is MySQL only then. I was thinking as much but didn't want to go read the ANSI SQL docs (wherever they live) 😅

    EDIT: StackOverflow suggests this might be the equivalent syntax?

    UPDATE patchwork_event event
SET
    event.cover_id = event.patch_id,
    event.cover_comment_id = cover_comment.id,
    event.category = 'cover-comment-created',
    event.patch_id = NULL,
FROM patchwork_patchcomment patch_comment
    INNER JOIN patchwork_covercomment cover_comment
    ON patch_comment.msgid = cover_comment.msgid
WHERE
    patch_comment.id = event.patch_comment_id
    AND event.patch_id IN (
        SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
    );

    ?

  16. alialnu commented on Sep 27, 2023

    @alialnu
    alialnu
    on Sep 27, 2023
    ContributorAuthor

    Sorry for the late response @stephenfin, I missed your last suggestion.

    The suggested query failed with:

    ERROR:  column "event" of relation "patchwork_event" does not exist
LINE 3:     event.cover_id = event.patch_id,

    I got it to work with these changes:

    UPDATE patchwork_event AS event
SET
    cover_id = event.patch_id,
    cover_comment_id = cover_comment.id,
    category = 'cover-comment-created',
    patch_id = NULL
FROM patchwork_patchcomment patch_comment
INNER JOIN patchwork_covercomment cover_comment
    ON patch_comment.msgid = cover_comment.msgid
WHERE
    patch_comment.id = event.patch_comment_id
    AND event.patch_id IN (
        SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
    UPDATE 26

    Removing duplicate patch comments still fails though:

    DELETE FROM patchwork_patchcomment AS patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
    ERROR:  update or delete on table "patchwork_patchcomment" violates foreign key constraint "patchwork_event_patch_comment_id_4123011e_fk_patchwork" on table "patchwork_event"
DETAIL:  Key (id)=(162805) is still referenced from table "patchwork_event".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugdatabaseIssues with the database or database migrations

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions