Verified
Status Update
No update yet.
Comments
lu...@gmail.com
This is a P1 because failing to load the External-Ids cache means you can't do anything with Gerrit: it is essential an outage.
lu...@gmail.com
It looks like a regression since https://gerrit-review.googlesource.com/c/gerrit/+/328299 when the differential loading is the ONLY way to load external ids.
sy...@gmail.com
[Empty comment from Monorail migration]
hi...@google.com
> There is something wrong with the differential refactoring, but I don't quite understand the code.
The code takes the old state and applies all modifications from Git commits to it that have happened since.
I am surprised we haven't seen this since 2019 in our installations and am wondering if there is something special with the external IDs/keys you use.
> Unclear, but it looks like a user added two external-ids within the same second
Same-second should not be a problem, we really just do a Git diff, so walk commits one-by-one.
The two SHA1s of the extId files are different, are redacted-1/2 different as well. Could this be a upper/lower casing issue? The naming of files (git notes style) should prevent duplicate keys in the storage. When you can rebuild from scratch, that means that the full-parser works fine. So I am wondering why the differential loader would in this case come to a different conclusion.
What form does redacted 1/2 have? Could this confuse the external ID key parser?
The code takes the old state and applies all modifications from Git commits to it that have happened since.
I am surprised we haven't seen this since 2019 in our installations and am wondering if there is something special with the external IDs/keys you use.
> Unclear, but it looks like a user added two external-ids within the same second
Same-second should not be a problem, we really just do a Git diff, so walk commits one-by-one.
The two SHA1s of the extId files are different, are redacted-1/2 different as well. Could this be a upper/lower casing issue? The naming of files (git notes style) should prevent duplicate keys in the storage. When you can rebuild from scratch, that means that the full-parser works fine. So I am wondering why the differential loader would in this case come to a different conclusion.
What form does redacted 1/2 have? Could this confuse the external ID key parser?
sy...@gmail.com
> The code takes the old state and applies all modifications from Git commits to it that have happened since.
This is the first time this happens for us too.
> I am surprised we haven't seen this since 2019 in our installations and am wondering if there is something special with the external IDs/keys you use.
We do use the github plugin, not sure if this plays a role.
> Unclear, but it looks like a user added two external-ids within the same second
Same-second should not be a problem, we really just do a Git diff, so walk commits one-by-one.
> The two SHA1s of the extId files are different, are redacted-1/2 different as well. Could this be a upper/lower casing issue? The naming of files (git notes style) should prevent duplicate keys in the storage. When you can rebuild from scratch, that means that the full-parser works fine. So I am wondering why the differential loader would in this case come to a different conclusion.
> What form does redacted 1/2 have? Could this confuse the external ID key parser?
They take the form of:
```
external:github_oauth:oauth_key_20221012:ffpM02W3p3Iv0yHnJsG6ErIKkY9J90GGH5GzqxpW+snvYGplACjIlocplLTaTVk9
```
I will try to reproduce it and report back my results in this card
Nothing special, as far as I can see.
This is the first time this happens for us too.
> I am surprised we haven't seen this since 2019 in our installations and am wondering if there is something special with the external IDs/keys you use.
We do use the github plugin, not sure if this plays a role.
> Unclear, but it looks like a user added two external-ids within the same second
Same-second should not be a problem, we really just do a Git diff, so walk commits one-by-one.
> The two SHA1s of the extId files are different, are redacted-1/2 different as well. Could this be a upper/lower casing issue? The naming of files (git notes style) should prevent duplicate keys in the storage. When you can rebuild from scratch, that means that the full-parser works fine. So I am wondering why the differential loader would in this case come to a different conclusion.
> What form does redacted 1/2 have? Could this confuse the external ID key parser?
They take the form of:
```
external:github_oauth:oauth_key_20221012:ffpM02W3p3Iv0yHnJsG6ErIKkY9J90GGH5GzqxpW+snvYGplACjIlocplLTaTVk9
```
I will try to reproduce it and report back my results in this card
Nothing special, as far as I can see.
sy...@gmail.com
[Empty comment from Monorail migration]
hi...@google.com
> I will try to reproduce it and report back my results in this card
Thanks!
Thanks!
gi...@appspot.gserviceaccount.com
The following revision refers to this bug:
https://gerrit.googlesource.com/gerrit/+/cc386031455272cc57b20174469353677e279d9c
commit cc386031455272cc57b20174469353677e279d9c
Author: Darius Jokilehto <dariusjokilehto+os@gmail.com>
Date: Fri Apr 21 13:53:45 2023
Log external ID differential cache loader failure
We've investigated this issue and tried different ways of getting this
to fail, but haven't been successful.
We've investigated the algorithm, but haven't found any faults. We also
considered concurrency, but as we lock the cache before calling the
differential loader, we're not sure how that could be the issue.
We've managed to recreate the issue by fudging the input to the private
method, where we pass a key to `additions` that is already present in
the cache, but we don't see how that would be possible from the public
`load` entrypoint.
In order to continue the investigation we require more information, we
have opted to add some further logs when the differential cache loader
errors.
We now log the tip of the external ID cache as well as the external ID
ref on disk. We also log the set of additions, to confirm a key clash.
Bug:https://crbug.com/gerrit/16384
Release-Notes: skip
Change-Id: I3836f28285b4fdcfd2e26720bdbf82cf046960c7
[modify]https://gerrit.googlesource.com/gerrit/+/cc386031455272cc57b20174469353677e279d9c/java/com/google/gerrit/server/account/externalids/ExternalIdCacheLoader.java
commit cc386031455272cc57b20174469353677e279d9c
Author: Darius Jokilehto <dariusjokilehto+os@gmail.com>
Date: Fri Apr 21 13:53:45 2023
Log external ID differential cache loader failure
We've investigated this issue and tried different ways of getting this
to fail, but haven't been successful.
We've investigated the algorithm, but haven't found any faults. We also
considered concurrency, but as we lock the cache before calling the
differential loader, we're not sure how that could be the issue.
We've managed to recreate the issue by fudging the input to the private
method, where we pass a key to `additions` that is already present in
the cache, but we don't see how that would be possible from the public
`load` entrypoint.
In order to continue the investigation we require more information, we
have opted to add some further logs when the differential cache loader
errors.
We now log the tip of the external ID cache as well as the external ID
ref on disk. We also log the set of additions, to confirm a key clash.
Bug:
Release-Notes: skip
Change-Id: I3836f28285b4fdcfd2e26720bdbf82cf046960c7
[modify]
lu...@gmail.com
[Comment Deleted]
lu...@gmail.com
This has happened again today, causing one whole site to go down immediately:
[2023-06-06T07:44:56.651Z] [HTTP GET /login/%2Flogin%2F%252Fdashboard%252Fself?final=true (N/A from <redacted>)] ERROR com.google.gerrit.server.account.externalids.ExternalIdCacheLoader : Failed to load external ID cache. Repository ref is d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7, cache ref is d9a8ea477ea7e6e9581559ec8ff651e0792587c9, additions are [36e02ce64b1f2440461b759cf9c91989881d1692]
java.lang.IllegalArgumentException: Multiple entries with same key: <redacted>=[externalId "<redacted>"]
accountId = 1019048
and <redacted>=[externalId "<redacted>"]
accountId = 1019048
at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:211)
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:205)
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:146)
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:109)
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:390)
at com.google.gerrit.server.account.externalids.ExternalIdCacheLoader.buildAllExternalIds(ExternalIdCacheLoader.java:267)
at com.google.gerrit.server.account.externalids.ExternalIdCacheLoader.load(ExternalIdCacheLoader.java:191)
at com.google.gerrit.server.account.externalids.ExternalIdCacheImpl.get(ExternalIdCacheImpl.java:118)
at com.google.gerrit.server.account.externalids.ExternalIdCacheImpl.get(ExternalIdCacheImpl.java:88)
at com.google.gerrit.server.account.externalids.ExternalIdCacheImpl.byAccount(ExternalIdCacheImpl.java:59)
at com.google.gerrit.server.account.externalids.ExternalIds.byAccount(ExternalIds.java:85)
at com.google.gerrit.server.account.AccountState.forCachedAccount(AccountState.java:133)
at com.google.gerrit.server.account.AccountCacheImpl.get(AccountCacheImpl.java:146)
at com.google.gerrit.server.account.AccountCacheImpl.get(AccountCacheImpl.java:106)
at com.google.gerrit.httpd.CacheBasedWebSession.checkAccountStatus(CacheBasedWebSession.java:230)
at com.google.gerrit.httpd.CacheBasedWebSession.login(CacheBasedWebSession.java:193)
at com.google.gerrit.httpd.auth.container.HttpLoginServlet.doGet(HttpLoginServlet.java:155)
The extra logging added displays:
ERROR com.google.gerrit.server.account.externalids.ExternalIdCacheLoader : Failed to load external ID cache. Repository ref is d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7, cache ref is d9a8ea477ea7e6e9581559ec8ff651e0792587c9, additions are [36e02ce64b1f2440461b759cf9c91989881d1692]
The commits are:
commit d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7
Date: Tue Jun 6 07:44:54 2023 +0000
|
|
commit d9a8ea477ea7e6e9581559ec8ff651e0792587c9
Date: Tue Jun 6 07:44:53 2023 +0000
The SHA1=36e02ce64b1f2440461b759cf9c91989881d1692 doesn't exist:
$ git cat-file -p 36e02ce64b1f2440461b759cf9c91989881d1692
fatal: Not a valid object name 36e02ce64b1f2440461b759cf9c91989881d1692
Basically, when it tries to add the external-id of the commit d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7, it creates the same id twice and therefore the key is duplicated.
[2023-06-06T07:44:56.651Z] [HTTP GET /login/%2Flogin%2F%252Fdashboard%252Fself?final=true (N/A from <redacted>)] ERROR com.google.gerrit.server.account.externalids.ExternalIdCacheLoader : Failed to load external ID cache. Repository ref is d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7, cache ref is d9a8ea477ea7e6e9581559ec8ff651e0792587c9, additions are [36e02ce64b1f2440461b759cf9c91989881d1692]
java.lang.IllegalArgumentException: Multiple entries with same key: <redacted>=[externalId "<redacted>"]
accountId = 1019048
and <redacted>=[externalId "<redacted>"]
accountId = 1019048
at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:211)
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:205)
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:146)
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:109)
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:390)
at com.google.gerrit.server.account.externalids.ExternalIdCacheLoader.buildAllExternalIds(ExternalIdCacheLoader.java:267)
at com.google.gerrit.server.account.externalids.ExternalIdCacheLoader.load(ExternalIdCacheLoader.java:191)
at com.google.gerrit.server.account.externalids.ExternalIdCacheImpl.get(ExternalIdCacheImpl.java:118)
at com.google.gerrit.server.account.externalids.ExternalIdCacheImpl.get(ExternalIdCacheImpl.java:88)
at com.google.gerrit.server.account.externalids.ExternalIdCacheImpl.byAccount(ExternalIdCacheImpl.java:59)
at com.google.gerrit.server.account.externalids.ExternalIds.byAccount(ExternalIds.java:85)
at com.google.gerrit.server.account.AccountState.forCachedAccount(AccountState.java:133)
at com.google.gerrit.server.account.AccountCacheImpl.get(AccountCacheImpl.java:146)
at com.google.gerrit.server.account.AccountCacheImpl.get(AccountCacheImpl.java:106)
at com.google.gerrit.httpd.CacheBasedWebSession.checkAccountStatus(CacheBasedWebSession.java:230)
at com.google.gerrit.httpd.CacheBasedWebSession.login(CacheBasedWebSession.java:193)
at com.google.gerrit.httpd.auth.container.HttpLoginServlet.doGet(HttpLoginServlet.java:155)
The extra logging added displays:
ERROR com.google.gerrit.server.account.externalids.ExternalIdCacheLoader : Failed to load external ID cache. Repository ref is d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7, cache ref is d9a8ea477ea7e6e9581559ec8ff651e0792587c9, additions are [36e02ce64b1f2440461b759cf9c91989881d1692]
The commits are:
commit d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7
Date: Tue Jun 6 07:44:54 2023 +0000
|
|
commit d9a8ea477ea7e6e9581559ec8ff651e0792587c9
Date: Tue Jun 6 07:44:53 2023 +0000
The SHA1=36e02ce64b1f2440461b759cf9c91989881d1692 doesn't exist:
$ git cat-file -p 36e02ce64b1f2440461b759cf9c91989881d1692
fatal: Not a valid object name 36e02ce64b1f2440461b759cf9c91989881d1692
Basically, when it tries to add the external-id of the commit d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7, it creates the same id twice and therefore the key is duplicated.
lu...@gmail.com
The duplicated addition is actually the one included in the tip of the All-Users:refs/meta/external-ids
commit d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7
Date: Tue Jun 6 07:44:54 2023 +0000
Link External ID
diff --git a/36/e0/2ce64b1f2440461b759cf9c91989881d1692
new file mode 100644
index 0000000000000..44f8f4608519e
--- /dev/null
+++
@@ -0,0 +1,2 @@
+[externalId "<redacted>"]
+ accountId = 1019048
I see also that there were previous attempts to update All-Users refs/meta/external-ids, which failed because of concurrency. Then the client retried automatically and boom.
We have seen already that JGit may return a failure to update the repository, even if it actually succeeded: that is due to concurrency in updating the packed-refs.
If we *think* to have failed the operation but actually we did not, we may end up messing up the in-memory cache because of the re-processing of the entity that we previously failed to insert.
Example:
Ext-Id1 => attempt to insert into All-Users: failed on JGit but not on disk. The key Ext-Id1 is stored in cache
Ext-Id1 => retry to insert into All-Users. The key Ext-Id1 is the entry to add BUT it already exists in cache => BOOM.
Basically, the differential loading is not idempotent and therefore will endemically fail in case of concurrency / retry to insert the same entry again, because of concurrency on the JGit ref-update.
That corresponds to the hypothesis formulated by Darius, that somehow the cache was already containing the key (somehow) that was received by the new commit, which is compatible with a retry logic.
@Patrick at this point in time, the fix seems quite easy: just make the additional of external-ids idem-potent. What do you think?
commit d95434d1ee9ccef7b8949685abb2e6b23e9cb9a7
Date: Tue Jun 6 07:44:54 2023 +0000
Link External ID
diff --git a/36/e0/2ce64b1f2440461b759cf9c91989881d1692
new file mode 100644
index 0000000000000..44f8f4608519e
--- /dev/null
+++
@@ -0,0 +1,2 @@
+[externalId "<redacted>"]
+ accountId = 1019048
I see also that there were previous attempts to update All-Users refs/meta/external-ids, which failed because of concurrency. Then the client retried automatically and boom.
We have seen already that JGit may return a failure to update the repository, even if it actually succeeded: that is due to concurrency in updating the packed-refs.
If we *think* to have failed the operation but actually we did not, we may end up messing up the in-memory cache because of the re-processing of the entity that we previously failed to insert.
Example:
Ext-Id1 => attempt to insert into All-Users: failed on JGit but not on disk. The key Ext-Id1 is stored in cache
Ext-Id1 => retry to insert into All-Users. The key Ext-Id1 is the entry to add BUT it already exists in cache => BOOM.
Basically, the differential loading is not idempotent and therefore will endemically fail in case of concurrency / retry to insert the same entry again, because of concurrency on the JGit ref-update.
That corresponds to the hypothesis formulated by Darius, that somehow the cache was already containing the key (somehow) that was received by the new commit, which is compatible with a retry logic.
@Patrick at this point in time, the fix seems quite easy: just make the additional of external-ids idem-potent. What do you think?
hi...@google.com
That sounds good to me.
gi...@appspot.gserviceaccount.com
The following revision refers to this bug:
https://gerrit.googlesource.com/gerrit/+/7974adfcbe92e903767e678c0629d2b46fdad026
commit 7974adfcbe92e903767e678c0629d2b46fdad026
Author: Fabio Ponciroli <ponch78@gmail.com>
Date: Wed Jun 07 10:19:09 2023
Fix ExternalId differential loading when cache is inconsistent
JGit may return a failure to update a repository, even if it actually
succeeded.
This can create an issue when loading externalIds, since
we would retry to insert the same value into the cache twice,
due to the retry performed by the client, leading the cache
loader to consistently blow up.
For example:
Ext-Id1:
* attempt to insert into All-Users: failed on JGit but not on disk [1]
* the key Ext-Id1 is stored in cache
Ext-Id1:
* retry to insert into All-Users, because of JGit failure
* try to add key Ext-Id1 entry BUT it already exists in cache => BOOM
Make differential loading idempotent to avoid failures due to retry to
insert the same entry again.
Make "buildAllExternalIds" package-private for testing visibility.
[1]:https://bugs.eclipse.org/bugs/show_bug.cgi?id=582044
Bug:https://crbug.com/gerrit/16384
Release-Notes: Fix ExternalId differential loading when cache is inconsistent
Change-Id: Iba6d538451994045b4455829d995dadd38866332
[modify]https://gerrit.googlesource.com/gerrit/+/7974adfcbe92e903767e678c0629d2b46fdad026/java/com/google/gerrit/server/account/externalids/ExternalIdCacheLoader.java
[modify]https://gerrit.googlesource.com/gerrit/+/7974adfcbe92e903767e678c0629d2b46fdad026/javatests/com/google/gerrit/server/account/externalids/ExternalIDCacheLoaderTest.java
commit 7974adfcbe92e903767e678c0629d2b46fdad026
Author: Fabio Ponciroli <ponch78@gmail.com>
Date: Wed Jun 07 10:19:09 2023
Fix ExternalId differential loading when cache is inconsistent
JGit may return a failure to update a repository, even if it actually
succeeded.
This can create an issue when loading externalIds, since
we would retry to insert the same value into the cache twice,
due to the retry performed by the client, leading the cache
loader to consistently blow up.
For example:
Ext-Id1:
* attempt to insert into All-Users: failed on JGit but not on disk [1]
* the key Ext-Id1 is stored in cache
Ext-Id1:
* retry to insert into All-Users, because of JGit failure
* try to add key Ext-Id1 entry BUT it already exists in cache => BOOM
Make differential loading idempotent to avoid failures due to retry to
insert the same entry again.
Make "buildAllExternalIds" package-private for testing visibility.
[1]:
Bug:
Release-Notes: Fix ExternalId differential loading when cache is inconsistent
Change-Id: Iba6d538451994045b4455829d995dadd38866332
[modify]
[modify]
gi...@appspot.gserviceaccount.com
The following revision refers to this bug:
https://gerrit.googlesource.com/gerrit/+/82896ef44ca8b7f29ea25ff86e39090af3791ece
commit 82896ef44ca8b7f29ea25ff86e39090af3791ece
Author: Fabio Ponciroli <ponch78@gmail.com>
Date: Wed Jun 07 10:19:09 2023
Fix ExternalId differential loading when cache is inconsistent
JGit may return a failure to update a repository, even if it actually
succeeded.
This can create an issue when loading externalIds, since
we would retry to insert the same value into the cache twice,
due to the retry performed by the client, leading the cache
loader to consistently blow up.
For example:
Ext-Id1:
* attempt to insert into All-Users: failed on JGit but not on disk [1]
* the key Ext-Id1 is stored in cache
Ext-Id1:
* retry to insert into All-Users, because of JGit failure
* try to add key Ext-Id1 entry BUT it already exists in cache => BOOM
Make differential loading idempotent to avoid failures due to retry to
insert the same entry again.
Make "buildAllExternalIds" package-private for testing visibility.
[1]:https://bugs.eclipse.org/bugs/show_bug.cgi?id=582044
Bug:https://crbug.com/gerrit/16384
Release-Notes: Fix ExternalId differential loading when cache is inconsistent
Change-Id: Iba6d538451994045b4455829d995dadd38866332
[modify]https://gerrit.googlesource.com/gerrit/+/82896ef44ca8b7f29ea25ff86e39090af3791ece/java/com/google/gerrit/server/account/externalids/ExternalIdCacheLoader.java
[modify]https://gerrit.googlesource.com/gerrit/+/82896ef44ca8b7f29ea25ff86e39090af3791ece/javatests/com/google/gerrit/server/account/externalids/ExternalIDCacheLoaderTest.java
commit 82896ef44ca8b7f29ea25ff86e39090af3791ece
Author: Fabio Ponciroli <ponch78@gmail.com>
Date: Wed Jun 07 10:19:09 2023
Fix ExternalId differential loading when cache is inconsistent
JGit may return a failure to update a repository, even if it actually
succeeded.
This can create an issue when loading externalIds, since
we would retry to insert the same value into the cache twice,
due to the retry performed by the client, leading the cache
loader to consistently blow up.
For example:
Ext-Id1:
* attempt to insert into All-Users: failed on JGit but not on disk [1]
* the key Ext-Id1 is stored in cache
Ext-Id1:
* retry to insert into All-Users, because of JGit failure
* try to add key Ext-Id1 entry BUT it already exists in cache => BOOM
Make differential loading idempotent to avoid failures due to retry to
insert the same entry again.
Make "buildAllExternalIds" package-private for testing visibility.
[1]:
Bug:
Release-Notes: Fix ExternalId differential loading when cache is inconsistent
Change-Id: Iba6d538451994045b4455829d995dadd38866332
[modify]
[modify]
lu...@gmail.com
[Empty comment from Monorail migration]
Description
*************************************************************************
*** !!!! THIS BUG TRACKER IS FOR GERRIT CODE REVIEW !!!!
*** Do not submit bugs for chrome/android and issues with your company's
*** Gerrit setup here. Those issues belong in different issue trackers.
*************************************************************************
What steps will reproduce the problem?
1. Unclear, but it looks like a user added two external-ids within the same second
commit b797f5a19c10bc33c38ff0e26fa1b019551d591esupport@gerrithub.io
Author: Gerrit Code Review
Date: Thu Nov 3 15:23:02 2022 +0000
diff --git a/23/76/d6cd86a131c02ba3954aed8352a8263d3858
new file mode 100644
index 000000000000..61e0df1996e1
--- /dev/null
+++
@@ -0,0 +1,2 @@
+[externalId "external:github_oauth:oauth_key_20221012:<redacted-2>"]
commit 338fa47ea4e8ff35be654281defb13758b2d46dasupport@gerrithub.io
Author: Gerrit Code Review
Date: Thu Nov 3 15:23:02 2022 +0000
diff --git a/0e/8b/226b2afad86ff2c6cdbf0b5fc212e7e71e19 b/0e/8b/226b2afad86ff2c6cdbf0b5fc212e7e71e19
new file mode 100644
index 000000000000..0667d5da0ff5
--- /dev/null
+++ b/0e/8b/226b2afad86ff2c6cdbf0b5fc212e7e71e19
@@ -0,0 +1,2 @@
+[externalId "external:github_oauth:oauth_key_20221012:<redacted-1>"]
What is the expected output?
The external_ids_map cache loading is performing the correct differential loading
What do you see instead?
The external_ids_map adds the key <redacted-2> twice and then fails constantly with:
[2022-11-03T15:23:03.720Z] [HTTP-2149362] WARN org.eclipse.jetty.server.HttpChannel : /login/%2Fq%2Fproject%3ATelecominfraproject%25252Foopt-gnpy
java.lang.IllegalArgumentException: Multiple entries with same key: external:github_oauth:oauth_key_20221012:<redactd-2>=[externalId "external:github_oauth:oauth_key_20221012:<redactd-2>"]
accountId = 1017751
and external:github_oauth:oauth_key_20221012:<redactd-2>=[externalId "<redactd-2>"]
accountId = 1017751
Please provide any additional information below.
There is something wrong with the differential refactoring, but I don't quite understand the code.