Is my queue table implementation race condition safe?
7
votes
4
answers
1108
views
Hello people smarter than me! I've created a sort-of-a-queue table system, but it seems too simple to be safe from race conditions. Am I missing something or is the following race condition safe?
The Schema
-
I have a table, let's call it
ProductQueue
:
CREATE TABLE dbo.ProductQueue
(
SerialId BIGINT PRIMARY KEY,
QueuedDateTime DATETIME NOT NULL -- Only using this for reference, no functionality is tied to it
);
I have procedure for adding to the queue called AddToProductQueue
:
CREATE PROCEDURE dbo.AddToProductQueue (@SerialId BIGINT)
AS
BEGIN
INSERT INTO dbo.ProductQueue (SerialId, QueuedDateTime)
OUTPUT Inserted.SerialId
SELECT @SerialId, GETDATE();
END
I also have a procedure for removing from the queue called RemoveFromProductQueue
:
CREATE PROCEDURE dbo.RemoveFromProductQueue (@SerialId BIGINT)
AS
BEGIN
DELETE FROM dbo.ProductQueue
OUTPUT Deleted.SerialId
WHERE SerialId = @SerialId;
END
Note, SerialId
is globally unique for a Product
in the source database / system. I.e. no two instances of a Product
can ever have the same SerialId
. That's the extent of it on the database side.
The Workflow
-
- I have an application process that runs hourly.
- That process gets a variable list of SerialIds
from the source system.
- It iteratively calls the AddToProductQueue
procedure on each SerialId
in its list.
- If the procedure tries to insert a SerialId
that exists in the ProductQueue
table already, it throws a primary key violation error, and the application process catches that error and skips that SerialId
.
- Otherwise, the procedure successfully adds that SerialId
to the ProductQueue
table and returns it back to the application process.
- The application process then adds that successfully queued SerialId
to a separate list.
- After the application process finishes iterating its list of all candidate SerialIds
to enqueue, it then iterates its new list of successfully queued SerialIds
and does external work on them, in a **separate thread** per SerialId
. (This work is all unrelated to the database.)
- Finally, as each thread finishes its external work, the last step in that asynchronous thread is to remove that SerialId
from the ProductQueue
table by calling the RemoveFromProductQueue
procedure. (Note that a new database context object is instantiated and a new connection is created for each asynchronous call to this procedure, so that it is thread-safe on the application side.)
Additional Information
-
- There aren't any indexes on the ProductQueue
table, and it'll never have more than 1,000 rows in it at one time. (Actually, most times it'll literally only have a couple of rows.)
- The same SerialId
can become a candidate again to be re-added to the queue table on a future execution of the application process.
- There are no safe guards from preventing a second instance of the application process from concurrently running, either by accident or if the first instance took more than 1 hour to run, etc. (This is the concurrent part I'm most concerned about.)
- The transaction isolation level of the database (and connection being made) where the queue table and procedures live is the default isolation level of Read Committed
.
Potential Problems
-
- The running instance of the application process crashes in an unhandled way, leaving SerialIds
stuck in the queue table. This is acceptable for the business needs, and we plan to have exception reports to help us manually remediate this case.
- The application process gets executed multiple times concurrently and grabs some of the same SerialIds
between instances in their initial source lists. I can't think of any negative ramifications of this case yet, since the enqueuing procedure is atomic, and the actual list of SerialIds
that the application process will work on should be self-contained due to that atomic enqueuing procedure. We don't care which instance of the application process actually processes each SerialId
as long as the same SerialId
isn't processed concurrently by both process instances.
Asked by J.D.
(40893 rep)
Mar 13, 2024, 09:46 PM
Last activity: Mar 16, 2024, 03:03 PM
Last activity: Mar 16, 2024, 03:03 PM