Database Administrators
Q&A for database professionals who wish to improve their database skills
Latest Questions
2
votes
1
answers
158
views
Reduce Parallelism Without Setting MAXDOP
I am trying to reduce parallelism in my query without explicitly setting MAXDOP. I have been informed that MAXDOP can hinder throughput as the platform grows and query executions increase (happy to be corrected on this). The following query has been observed hitting MAXDOP 8, causing spikes of worke...
I am trying to reduce parallelism in my query without explicitly setting MAXDOP. I have been informed that MAXDOP can hinder throughput as the platform grows and query executions increase (happy to be corrected on this).
The following query has been observed hitting MAXDOP 8, causing spikes of workers in our Azure SQL database. The result is the error "the request limit for the database is 800 and has been reached"
The following has been applied to try and make this query less parallel:
1. Split the queries into #ErIds and #BaseTable
2. Change a big CASE statement into individual IFs for the sort
If anyone has any ideas on how to reduce parallelism that would be really appreciated.
I can provide more info if needed.
Thank you.
```
declare @uniqueIds dbo.GuidList
, @ids dbo.IntegerList
, @searchTerm nvarchar(max) = null
, @pageNum int
, @pageSize int
, @sortBy nvarchar(50) = 'Name'
, @sortDescending bit = 0
set nocount on;
-- creating these local variables to avoid parameter sniffing
declare @pageNumLocal int = case
when @pageNum is null
or @pageNum <= 0
then 0
else @pageNum
end
, @pageSizeLocal int = isnull(@pageSize, 999999) -- If pageSize is null, return all records
, @sortByLocal nvarchar(50) = isnull(@sortBy, 'Name')
, @sortOrderLocal nvarchar(20) = 'asc'
, @searchTermLocal nvarchar(50) = '%' + lower(trim(isnull(@searchTerm, ''))) + '%';
if @sortDescending = 1
set @sortOrderLocal = 'desc'
else
set @sortOrderLocal = 'asc';
create table #ErIds(id integer primary key);
insert into #ErIds
select distinct e.Id
from [TableA] e
where e.IsSoftDeleted = 0
and (
e.Id in (
select intVal
from @ids
)
or e.UniqueId in (
select guidVal
from @uniqueIds
)
)
create table #BaseTable (
Id int,
UniqueId uniqueidentifier,
[Name] nvarchar(200),
EmployeeCount int,
ProcessorName nvarchar(255)
);
insert into #BaseTable
select e.Id,
e.UniqueId,
e.[Name],
e.EmployeeCount,
ProcessorName = isnull(u.FirstName + ' ' + u.LastName, u.EmailAddress),
from #ErIds erids
inner join [TableA] e
on erids.id = e.id
left join [TableB] bs
on bs.EmployerId = e.Id
left join [TableC] u
on u.Id = bs.ProcessorId
where
(
isnull(@searchTermLocal, '') = ''
or lower(trim(e.[Name])) like @searchTermLocal
or lower(trim(isnull(u.FirstName + ' ' + u.LastName, u.EmailAddress))) like @searchTermLocal
or lower(trim(convert(nvarchar(36), e.UniqueId))) like @searchTermLocal
)
if @sortByLocal = 'Name'
begin
if @sortOrderLocal = 'asc'
select * from #BaseTable order by [Name] asc offset @pageNumLocal * @pageSizeLocal rows fetch next @pageSizeLocal rows only;
else
select * from #BaseTable order by [Name] desc offset @pageNumLocal * @pageSizeLocal rows fetch next @pageSizeLocal rows only;
end
else if @sortByLocal = 'ProcessorName'
begin
if @sortOrderLocal = 'asc'
select * from #BaseTable order by ProcessorName asc offset @pageNumLocal * @pageSizeLocal rows fetch next @pageSizeLocal rows only;
else
select * from #BaseTable order by ProcessorName desc offset @pageNumLocal * @pageSizeLocal rows fetch next @pageSizeLocal rows only;
end
else if @sortByLocal = 'EmployeeCount'
begin
if @sortOrderLocal = 'asc'
select * from #BaseTable order by EmployeeCount asc offset @pageNumLocal * @pageSizeLocal rows fetch next @pageSizeLocal rows only;
else
select * from #BaseTable order by EmployeeCount desc offset @pageNumLocal * @pageSizeLocal rows fetch next @pageSizeLocal rows only;
end
drop table #ErIds;
drop table #BaseTable;
SkelDave
(163 rep)
Jul 7, 2025, 09:25 AM
• Last activity: Jul 8, 2025, 07:45 PM
0
votes
1
answers
218
views
Should I change parallelism (MaxDop and Threshold)?
Seems like I have problems with parallelism because largest wait types are `CXCONSUMER` and `CXPACKET`. Server has 8 cores. So I am planning to bump up Cost Threshold to **50** and MaxDop to **4** Currently I have default values, which is * 5 - cost threshold for parallelism * 0 - max degree of para...
Seems like I have problems with parallelism because largest wait types are
Mostly, only a couple of databases are used intensively out of all databases we have on instance. Which makes me wonder whether I should implement those changes on a whole instance or only on a couple of databases.
CXCONSUMER
and CXPACKET
.
Server has 8 cores.
So I am planning to bump up
Cost Threshold to **50** and
MaxDop to **4**
Currently I have default values, which is
* 5 - cost threshold for parallelism
* 0 - max degree of parallelism

Serdia
(707 rep)
Jul 6, 2021, 10:27 PM
• Last activity: Jun 9, 2025, 01:08 AM
1
votes
2
answers
45
views
Unexpected occasional DEADLOCK when re-recreating database from split MySQL dump
To replace the content of a test database from a production database `mysqldump` output the following command was used: cat mysqldump-db-thisdate.sql | mysql -p ... mydb There has never been any issue with this command. However the DB grew a lot and this command takes several minutes. In order to re...
To replace the content of a test database from a production database
mysqldump
output the following command was used:
cat mysqldump-db-thisdate.sql | mysql -p ... mydb
There has never been any issue with this command.
However the DB grew a lot and this command takes several minutes.
In order to reduce this time, a Perl script was written that
- takes the mysqldump output as input
- creates a single file having all DROP TABLE
... CREATE TABLE
for each table
- run this drop-creation file on a single thread, before doing the tables feeding below
- creates as many files (see below) as there are tables (about 100 tables)
- makes a fork()
for each table file that is injected into the DB (all tables are dropped and created + fed in "parallel". table1..100
The DROP-CREATION file is something like
DROP TABLE IF EXISTS mytable1
;
CREATE TABLE mytable1
(
someid1
int NOT NULL,
...
PRIMARY KEY (someid1
)
) ENGINE=InnoDB AUTO_INCREMENT=111 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;
DROP TABLE IF EXISTS mytable2
;
CREATE TABLE mytable2
(
someid2
int NOT NULL,
...
PRIMARY KEY (someid2
)
) ENGINE=InnoDB AUTO_INCREMENT=222 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;
...
Each table file mytableI.sql
is like this, for instance for mytable1.sql
/*!40103 SET TIME_ZONE='+00:00' */;
SET foreign_key_checks = 0;
SET unique_checks = 0;
SET autocommit = 0;
START TRANSACTION;
LOCK TABLES mytable1
WRITE;
INSERT INTO mytable1
VALUES (...),(...),...;
UNLOCK TABLES;
COMMIT;
It's like doing, in parallel (pseudo code)
for each table 1 to 100 do
cat mytableI.sql | mysql -p ... mydb /* I is 1 ... 100 */
end for
This method works very well, and saves from 50% to 75% of the time compared to the simple cat whole-dump | mysql
usual method.
However, from time to time (maybe 1 / 10), doing this parallel method, mysql
throws an error
> Deadlock found when trying to get lock; try restarting transaction
It happens rarely, so just restarting the command is not a big deal.
But why? Each table is processed at once, foreign keys are not checked... Doesn't MySQL, thanks to "LOCK TABLES" (and other mechanisms) protect itself against deadlocks in this case?
*Addendum*: The mydb test database is not being accesses otherwise.
***edit testing other methods***
Trying to perform the DROP / CREATE operations in parallel, (each DROP / CREATE in the same thread, for each table), not even filling the tables with data, plenty of Deadlocks occur...
Could it be that MySQL does not handle very well DROP/CREATE operations performed simultaneously? (should be done by a single DB admin?)
Note:
"*simultaneously*" and "*in parallel*" meaning each thread has its own MySQL connection.
Déjà vu
(555 rep)
May 29, 2025, 07:29 AM
• Last activity: May 31, 2025, 12:15 PM
3
votes
3
answers
3593
views
What can I do about high CXPACKET waits?
We are experiencing some slowness on our MS SQL Server 2016 database, I have been using Brent Ozar's first aid kit to do some initial troubleshooting. I am seeing a high amounts of CXPACKET wait types, out of a 17.5 hour data sample we saw 99 hours of wait across our 10 CPU's, that 55.5%! I was hopi...
We are experiencing some slowness on our MS SQL Server 2016 database, I have been using Brent Ozar's first aid kit to do some initial troubleshooting.
I am seeing a high amounts of CXPACKET wait types, out of a 17.5 hour data sample we saw 99 hours of wait across our 10 CPU's, that 55.5%!
I was hoping someone here could confirm that we should be concerned about this number and resolve it asap. We have MAXDOP setting of 4 which is accurate to MS recommendations but our CTP is set to 5 which I believe needs to be changed to 50.
Just looking for clarification before I take this information to my boss, yes, I am new to database administration and yes I am looking at other wait types but this seems to be the most significant so far.
Cheers,
Josh
user3547606
(31 rep)
Jun 28, 2018, 03:55 AM
• Last activity: Apr 14, 2025, 10:14 PM
2
votes
0
answers
60
views
How to make PostgreSQL take advantage of SSD RAID?
I'm using a PG13 server as a feature DB. There is no else job on the server but this PG instance and a GPU based machine learning process. There is no online transaction, and the data lossing or incorrect are not trouble, I just need it run faster. Few days ago, with a lot of gentlemen's help in [my...
I'm using a PG13 server as a feature DB.
There is no else job on the server but this PG instance and a GPU based machine learning process.
There is no online transaction, and the data lossing or incorrect are not trouble, I just need it run faster.
Few days ago, with a lot of gentlemen's help in my another question , I have corrected some mis-configurations of my PG server.
But the throughput of the PG server delivering data still can **NOT** satisfy the reuqiement of the ML process. I need the PG server to produce data as fast as possible, or in other word, to load data from disk more aggressively.
There are two physical CPU(as two numa nodes), and each of them has 128 real cores(I disabled Super Threads in BIOS setup), 128GB memory(I splited 64GB dedicated for PG).
Since whole of CPU1 has been assigned dedicated for PG use, PG is configed with 128 parallel workers.
The result is **NOT** as my expecting, the throughput still lower.
The bandwidth of the SSD RAID is 10GB/s, but the actual reading/writing rate of PG is 100~200MB/s, only like a SATA hard disk.
I don't believe it has only so poor performance, because I did some experiments/observing as fowllowing:
1. Directly read/write raw file by means of
dd
command, and count the io rate. I even writed a C++ program that starts multi-threads
directly dump/load data to/from files on the RAID. The io rate can
easily continuously exceed 10GB/s viewed by iotop
command. So I
think that the configurations of the hardware, driver soft and the
OS(Debian12) should be ok;
2. By watching the output of top
/numastat
, I found that most PG workers mearly use very less of CPU/mem resources, even though there are 128 hungry clients awaiting data. If they(PG workers) are waiting for io, the RAID should be very busy. But I neven observed the io rate exceed 300MB/s when PG working. If they(PG workers) are doing some DB operations such as sorting/filtering/aggregating, should I see a higher usage percentage of CPU1? Neither io rate nor CPU usage are high, what are they doing when 128 parallel clients are running and constantly doing a simple query(with different filter arguments)?
3. I have tried to limit all processes of PG to run at a same numa node by means of numactl --cpunodebind 1 --membind 1 -- ...
, in order to prevent "CPU cache ping-pong" occursing between numa nodes. I also limited the client processes run only at CPU0, to prevent them from contention CPU1. But the throughput didn't improve.
4. I tried different values of option effective_io_concurrency
in postgresql.conf, such as 1, 256, 512, even 1000, but the results are similar.
How to make my PG server really busy up?
Forgive my ugly English, I hope I expressed things correctly. Thanks!
Leon
(411 rep)
Mar 26, 2025, 09:11 AM
• Last activity: Mar 26, 2025, 09:46 AM
0
votes
1
answers
946
views
Parallel insert and delete into staging tables
I work on a small system that receives quite a lot of data from multiple sources (hundreds of them). Depending on few factors I can get anywhere from few rows to a couple of thousands (in fact smallest message contains single row and largest contains at most five thousands, but if more data should b...
I work on a small system that receives quite a lot of data from multiple sources (hundreds of them). Depending on few factors I can get anywhere from few rows to a couple of thousands (in fact smallest message contains single row and largest contains at most five thousands, but if more data should be loaded then I get this data in few batches). Currently data is processed by few identical services. Each service load data into one of five staging tables and executes procedure that does the right thing - each staging table has different procedure associated with it, but it always boils down to changing unique identifiers (e.g. UUIDs) to proper database identifiers, inserting data into destination table and removing data from staging table. All procedures have this form:
INSERT INTO TARGET (A, B, C)
SELECT T1.ID, B, C FROM STAGING_TABLE
JOIN T1 ON A = T1.ID
DELETE FROM STAGING_TABLE
It is possible for all services to work on single staging (and destination) table in parallel. Currently this is done using snapshot isolation, but it is painfully obvious that for one reason or another we are losing data. What I mean by it is that there are messages that are properly processed by the services but all information from them is lost - we don't see records in the database. I can't prove that snapshot isolation is in responsible, but such incidents started to happen after snapshot isolation was introduced - which in turn was introduced after services loading data in parallel were introduced. Databases are currently far from my main field of expertise and I don't know why it happens, but it seems that snapshot isolation is the main culprit.
My question is: what is the lowest isolation level that can support this scenario? Is there better way to do it? I'm not fully aware what transaction isolation level was used earlier (when data was loaded by single service), but we never observed data loss. I tried (blindly) using "serializable" and "repeatable read", but "serializable" results in dropped messages due to deadlocks and "repeatable read" while seems to do the right thing (no data loss) also degrades performance to the level of serial writing.
EDIT:
Is it viable to load data using snapshot isolation, insert into temporary table (or table variable), THEN switch to some very permissive isolation level, insert data from temporary table into target table, revert to snapshot and delete data from staging table? If I read this correctly:
https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2005/ms173763(v=sql.90)
It should be possible, but I don't understand yet if any of this would have any effect in discussed case - target table is not read, only written in this scenario, and I *think* this means that write won't be any "more parallel" than under snapshot isolation. But maybe I'm wrong?
Note that we can't wait and for example load data from multiple sources into single staging table and then move it into target table. We're not aiming for real time, but we need data insert ASAP.
Jędrzej Dudkiewicz
(173 rep)
Jul 8, 2019, 05:17 PM
• Last activity: Feb 12, 2025, 10:38 AM
9
votes
2
answers
1199
views
In SQL Server, is parallelism per operator, or something else?
I work with a really old DBA who says a lot of weird stuff. Dude has an O'Reilly book that only has an amoeba on the cover. At lunch we were talking about parallelism, because our new server has 24 cores. He says that in a parallel plan, every operator gets DOP threads. So if you have MAXDOP 8 and y...
I work with a really old DBA who says a lot of weird stuff. Dude has an O'Reilly book that only has an amoeba on the cover.
At lunch we were talking about parallelism, because our new server has 24 cores. He says that in a parallel plan, every operator gets DOP threads. So if you have MAXDOP 8 and your query has 4 parallel operators it'll use 32 threads at once.
That doesn't seem right because you'd run out of threads really fast.
I also read that it might just be 8 for the whole query, which seems like too few.
Why Do I see more threads per SPID in sysprocesses than MAXDOP?
Are either of them right?
WorldStar SQL
(217 rep)
May 4, 2018, 12:53 AM
• Last activity: Feb 5, 2025, 10:04 PM
1
votes
3
answers
1201
views
How to Ensure a Query Runs in Parallel in PostgreSQL?
I have a PostgreSQL 16 database setup with the following configuration: ```sql -- DB Version: 16 -- OS Type: linux -- DB Type: oltp -- Total Memory (RAM): 64 GB -- CPUs num: 32 -- Connections num: 200 -- Data Storage: ssd show effective_cache_size; --48 GB show effective_io_concurrency; --200 show h...
I have a PostgreSQL 16 database setup with the following configuration:
-- DB Version: 16
-- OS Type: linux
-- DB Type: oltp
-- Total Memory (RAM): 64 GB
-- CPUs num: 32
-- Connections num: 200
-- Data Storage: ssd
show effective_cache_size; --48 GB
show effective_io_concurrency; --200
show huge_pages; --try
show maintenance_work_mem; --4GB
show max_connections; --200
show max_parallel_maintenance_workers; --4
show max_parallel_workers; --32
show max_parallel_workers_per_gather; --4
show max_wal_size; --8GB
show max_worker_processes; --32
show min_wal_size; --2GB
show parallel_setup_cost; --0.1
show random_page_cost; --1.1
show shared_buffers; --16GB
show wal_buffers; --16MB
show work_mem; --256MB
I am running a CREATE TABLE
query with multiple joins. Sometimes the query runs in parallel, but other times it does not. I want to ensure that the query runs in parallel or at least increase the chances of it running in parallel for that specific transaction session. It is also fine if parallel workers assigned are fewer than usual because without parallel workers the query is very slow.
The configuration below worked best for me for some time, but it caused an error similar to this [issue](https://stackoverflow.com/questions/64094389/postgresql-11-5-could-not-resize-shared-memory-segment-postgresql-xxx-to-yy) .
ALTER SYSTEM SET maintenance_work_mem = '8GB';
ALTER SYSTEM SET work_mem = '2GB';
ALTER SYSTEM SET effective_io_concurrency = '400';
ALTER SYSTEM SET max_parallel_workers_per_gather = '16';
ALTER SYSTEM SET wal_buffers = '1GB';
I am attaching a select query plan for reference: [query plan](https://pastebin.com/pE9k1Sfc) .
I want to maximize CPU utilization (currently less than 10%) and RAM utilization (currently less than 30%).
I am aware that force_parallel_mode
has been deprecated. How can I achieve consistent parallel execution for my queries in PostgreSQL?
Purushottam Nawale
(161 rep)
Jan 17, 2025, 09:04 PM
• Last activity: Jan 21, 2025, 06:07 PM
5
votes
3
answers
1446
views
A query submitted from different applications has differing DOP
I have a query which selects from multiple views and is fairly I/O heavy. If I execute this query using Management Studio it uses parallelism across most of the 16 CPUs and completes in under 10 seconds. However, when executing it from SpotFire (a product from Tibco) it uses only 1 CPU and can take...
I have a query which selects from multiple views and is fairly I/O heavy. If I execute this query using Management Studio it uses parallelism across most of the 16 CPUs and completes in under 10 seconds. However, when executing it from SpotFire (a product from Tibco) it uses only 1 CPU and can take hours to complete.
I am able to replicate the issue in Management Studio if I using the query hint **OPTION (MAXDOP 1)**.
I was thinking that it may be a SpotFire problem, but it uses the Microsoft JDBC driver for connection and I see no connection properties regarding parallelism. I have confirmed using Profiler that the queries from SpotFire and from Management Studio look exactly. the. same. The only difference appears to be parallel execution.
Can anyone offer insight as to why this might occur?
**!SOLVED! [kind of]**
I reran the SQL Profiler trace and found that the queries are indeed the same, but the way that they are executed is different.
**From SpotFire:**
declare @p1 int
set @p1=0
declare @p2 int
set @p2=0
declare @p7 int
set @p7=0
exec sp_cursorprepexec @p1 output,@p2 output,NULL,N'SELECT a, b, c, d FROM MyView',16,8193,@p7 output
select @p1, @p2, @p7
**From Management Studio:**
SELECT a, b, c, d FROM MyView
I then added two new events to SQL Profiler and reran the queries:
- Performance / Degree of Parallelism
- Performance / Showplan All
This showed the the query from SpotFire was using DOP 0 (BinaryData = 0x00000000) and from Management Studio was DOP 12 (BinaryData = 0x1200000). Next, the SHOWPLAN showed me that the SpotFire query was inserting the results from the view into a temporary table before returning the data to the client.
So why does it do this? Possibly because of the **sp_cursorprepexec** statement. But why should that cause the DOP to drop to 0? I don't know.
I think that the "solution" is going to be working with SpotFire and possibly tuning the JDBC connection string.
[Here is an article from MSDN regarding all connection parameters for JDBC.](http://msdn.microsoft.com/en-us/library/ms378988(v=sql.90).aspx)
[Here is another article from someone who had the same exact problem I have, also with no resolution.](http://social.msdn.microsoft.com/Forums/en-US/sqldataaccess/thread/f8cf63b3-951d-422f-b046-41bb3852642c/)
Josh Bond
(719 rep)
Jun 23, 2011, 05:01 PM
• Last activity: Jan 12, 2025, 05:32 AM
0
votes
1
answers
258
views
Query running successfully but much longer due to wait_event MessageQueueSend
I have a long running bug where some larger queries, sometimes run much much longer due to being stuck on wait_even MessageQueueSend. The difference can be anything from <100% to 1000s percent when compared to optimal run time, which sometimes also happens, so this is a benchmark. The problem seems...
I have a long running bug where some larger queries, sometimes run much much longer due to being stuck on wait_even MessageQueueSend. The difference can be anything from <100% to 1000s percent when compared to optimal run time, which sometimes also happens, so this is a benchmark.
The problem seems to not be related to query performance/optimization in itself. Earlier the query had several CTEs which I then removed. This decreased the happy-path processing from ~43s to ~23s, but it did not get rid of this problem. When it gets stuck it can be there for several minutes or more. The same query can run sometimes in 23s and sometimes in 15mins.
I've digged into this topic a little, it is seems to be something related to IPC and shared memory of processes comprising the query - screenshot attached. MessageQueueSend wait event signifies that the parallel worker processes are waiting to send bytes to a shared message queue. But I'm lost when it comes to why that may happen - shared message queue is full? main process is busy and cannot collect data from queue? - and what can be done about it.
I've tried adjusting postgresql.conf, and the common values related to performance are as follows:
- max_connections = 20
- shared_buffers = 2GB
- effective_cache_size = 6GB
- maintenance_work_mem = 1GB
- checkpoint_completion_target = 0.9
- wal_buffers = 16MB
- default_statistics_target = 500
- random_page_cost = 1.1
- effective_io_concurrency = 200
- work_mem = 26214kB
- huge_pages = off
- min_wal_size = 4GB
- max_wal_size = 16GB
- max_worker_processes = 4
- max_parallel_workers_per_gather = 2
- max_parallel_workers = 4
- max_parallel_maintenance_workers = 2
**Tech stack:**
- postgres v.16 on linux VM, 8gb of ram, 4 CPUs from cluster
- queries are triggered on database by dbt v1.9
- db purpose: data warehouse, running daily several larger queries
P.S.
I've searched stack exchange posts, stack overflow posts, google, asked multiple LLMs and spent many hours trying to figure it out, although I'm a junior, so that's that.

user20061
(1 rep)
Jan 7, 2025, 05:19 PM
• Last activity: Jan 8, 2025, 04:31 AM
0
votes
1
answers
334
views
Index creation using 1 core only
I've been working on an environment where every index creation is happening on a single core. Big or small indexes/tables. Even when I force `MAXDOP` in the creation it still ignores it and runs on a single core. * `MAXDOP` is set to 0. * Available cores are 24. * Cost threshold is set at 5. * The i...
I've been working on an environment where every index creation is happening on a single core. Big or small indexes/tables.
Even when I force
MAXDOP
in the creation it still ignores it and runs on a single core.
* MAXDOP
is set to 0.
* Available cores are 24.
* Cost threshold is set at 5.
* The index creation cost more than 1000.
* SQL configured memory is: 160GB, even though 128GB is max.
SQL Server running:
-none
Microsoft SQL Server 2019 (RTM-CU18) (KB5017593) - 15.0.4261.1 (X64)
Sep 12 2022 15:07:06
Copyright (C) 2019 Microsoft Corporation
Standard Edition (64-bit) on Windows Server 2019 Standard
10.0 (Build 17763: ) (Hypervisor)
Why is it not able to go parallel?
Firat Aygur
(53 rep)
Jul 26, 2023, 12:49 PM
• Last activity: Dec 22, 2024, 08:54 AM
0
votes
1
answers
121
views
How do "Parallel query processing on partitioned tables and indexes" and "Partitioned table parallelism" differ?
In plain English, "Parallel query processing on partitioned tables and indexes" and "Partitioned table parallelism" sound like the same thing. Yet, in SQL Server 2022, [Standard Edition has "Partitioned table parallelism"](https://learn.microsoft.com/en-us/sql/sql-server/editions-and-components-of-s...
In plain English, "Parallel query processing on partitioned tables and indexes" and "Partitioned table parallelism" sound like the same thing. Yet, in SQL Server 2022, [Standard Edition has "Partitioned table parallelism"](https://learn.microsoft.com/en-us/sql/sql-server/editions-and-components-of-sql-server-2022?view=sql-server-ver16#RDBMSSP) but not [Parallel query processing on partitioned tables and indexes](https://learn.microsoft.com/en-us/sql/sql-server/editions-and-components-of-sql-server-2022?view=sql-server-ver16#data-warehouse) . How do the two features differ?
I have made a reasonable effort to search through the documentation, but have not yet read all of the white papers. I have not been successful in learning what either feature is.
J. Mini
(1225 rep)
Dec 20, 2024, 10:04 PM
• Last activity: Dec 22, 2024, 08:48 AM
6
votes
1
answers
453
views
Why is the tipping point between a serial and parallel plan not exactly the point where the serial plan is costed less?
For the example data... ``` /*Quick and dirty generation of some rows of data*/ SELECT value as [orderid], 1 as [custid], 1 as [empid], 1 as [shipperid], getdate() as [orderdate], 'abcdefgh' as [filler] INTO dbo.Orders FROM generate_series(1,10000000) CREATE CLUSTERED INDEX [idx_cl_od] ON [dbo].[Ord...
For the example data...
/*Quick and dirty generation of some rows of data*/
SELECT value as [orderid],
1 as [custid],
1 as [empid],
1 as [shipperid],
getdate() as [orderdate],
'abcdefgh' as [filler]
INTO dbo.Orders
FROM generate_series(1,10000000)
CREATE CLUSTERED INDEX [idx_cl_od] ON [dbo].[Orders]
(
[orderdate] ASC
)
UPDATE STATISTICS dbo.Orders WITH FULLSCAN
And the following query
SELECT [orderid], [custid], [empid], [shipperid], [orderdate], [filler]
FROM dbo.Orders
WHERE orderid <=7601715 AND 1=1 /*Prevent simple parameterisation*/
Then on my dev machine (SQL Server 2022, DOP of 4) the IO cost of the clustered index scan is 46.8853
regardless of serial or parallel plan.
And the CPU cost of the scan is 11.0002
in the serial plan and 2.75004
in the parallel one
So I was expecting the tipping point between the plans to be when the parallelism operator exceeded 8.25016
(a threshold reached when estimated rows going into it is around 4.5 million). In reality at the point this actually occurs the cost for the gather streams operator is 13.0501
(around 3 million rows higher than I expected).
If SQL Server isn't using overall plan cost as the tipping point what *is* the actual logic?
(XML for estimated plans on pastetheplan )

Martin Smith
(87941 rep)
Dec 7, 2024, 11:22 AM
• Last activity: Dec 7, 2024, 04:09 PM
15
votes
3
answers
42733
views
High CXPACKET and LATCH_EX waits
I am having some performance issues with a data processing system which I am working on. I have gathered wait stats from a one hour peroid which show a large amount of CXPACKET and LATCH_EX wait events. The system consists of 3 processing SQL Servers which do alot of number crunching and calculation...
I am having some performance issues with a data processing system which I am working on. I have gathered wait stats from a one hour peroid which show a large amount of CXPACKET and LATCH_EX wait events.
The system consists of 3 processing SQL Servers which do alot of number crunching and calculations and then feed the data into a central cluster server. The processing servers can have up to 6 jobs running each at any one time. These wait stats are for the central cluster which I think is causing a bottlneck. The central cluster server has 16 cores and 64GB RAM. MAXDOP is set to 0.
I guess the CXPACKET is from the multiple parallel queries running however I am not sure what the LATCH_EX wait event is indicating. From what I have read this could be a non-buffer wait?
Can anyone suggest what the cause of these kind of waits stats would be and what course of action I should be taking to investigate the root cause of this performance issue?
The top query results are the total wait stats and the bottom query result is the stats over the 1 hour period

stumct
(191 rep)
Mar 6, 2014, 09:13 AM
• Last activity: Nov 14, 2024, 07:38 AM
1
votes
2
answers
293
views
Do parallel SELECTs make a transaction?
While running under `READ COMMITTED` (the pessimistic type, not RCSI), I've just seen an `UPDATE`'s IX lock deadlock with a `SELECT`'s shared lock on the `PAGE`-level. The `SELECT` ran with a parallel execution plan. Neither explicitly used a transaction. I understand that `UPDATE`s implicitly creat...
While running under
READ COMMITTED
(the pessimistic type, not RCSI), I've just seen an UPDATE
's IX lock deadlock with a SELECT
's shared lock on the PAGE
-level. The SELECT
ran with a parallel execution plan. Neither explicitly used a transaction. I understand that UPDATE
s implicitly create transactions. Does this suggest that parallel SELECT
s also create implicit transactions? Is this not true of non-parallel SELECT
s?
I'm finding this rather confusing, because none of the textbook examples of deadlocks show two transaction-less queries deadlocking.
J. Mini
(1225 rep)
Oct 17, 2024, 03:51 PM
• Last activity: Oct 18, 2024, 12:18 PM
0
votes
1
answers
1181
views
Turn on multi-threading in Oracle for ETL purposes
As you can see from this question, I was able to configure multi-threading in SQL Server and build a workflow in Informatica to multi-thread a job to move data faster into SQL Server. https://dba.stackexchange.com/questions/107966/multi-thread-informatica-connections-to-use-different-processors-on-t...
As you can see from this question, I was able to configure multi-threading in SQL Server and build a workflow in Informatica to multi-thread a job to move data faster into SQL Server.
https://dba.stackexchange.com/questions/107966/multi-thread-informatica-connections-to-use-different-processors-on-target-datab
Now I'd like to do the same thing in Oracle. Does anyone know the feature in Oracle that's comparable to the "Max Degree of Parallelism" setting in SQL Server? I'd like to move data on each core of the server in parallel.
I'm strictly dealing with
INSERT
statements. So DML.
JustBeingHelpful
(2116 rep)
Jan 11, 2016, 06:46 PM
• Last activity: Sep 20, 2024, 09:06 PM
6
votes
1
answers
696
views
Tale of two servers. What could be limiting queries to running on two cores? MDOP set to 16
I have two servers. A garbage picked, slapped together with duct tape and bubble gum. It is a mix of Dell R620 and Gen 8 HPE parts with 40 cores / 1.5 TB ram. Running ubuntu 22.04 and sql server 2022 Dev Version. Raid 10 3.7 TB SSD's. MDOP is set to 16. The other is a VM hosted on a triple head clus...
I have two servers.
A garbage picked, slapped together with duct tape and bubble gum. It is a mix of Dell R620 and Gen 8 HPE parts with 40 cores / 1.5 TB ram. Running ubuntu 22.04 and sql server 2022 Dev Version. Raid 10 3.7 TB SSD's. MDOP is set to 16.
The other is a VM hosted on a triple head cluster Xenon Gold's with ssd Nas the whole 9 yards.
Server 2019, sql server standard 2022. 128 gigs / 16 cores allocated. MDOP is also set to 16. Database was a backup / restore from the test server.
The main table houses live streamed data averaging 100-200 mil new records a day, currently approaching 30 billion rows with 4 columns. It is setup with a column store index and the data is sorted and populated via a timestamp field.
Doing a simple query takes seconds to minutes on the test server but minutes to almost an hour on the soon to be production server.
The test server shows all 16 cores being utilized. On the production server it will only utilize two cores. If I run multiple instances of the same query the production server will consume an additional set of two cores per query executed. This can be done until the CPU is at 100% utilization.
The query I am testing with is
select top 100 * from mlinkdatacsunixtime order by serialtime desc
On the test server it runs in 1 min to 1.5 min. On production it is 11 minutes.
I have checked:
VM socket vs Core count. 2 Sockets / 8 Cores each.
Verified sql server shows all 16 as online and visible.
Matched up all the options between the two sql instances.
Doubled checked server and DB MDOP settings.
Executed Query multiple times to try to load the table in memory. Table is about 50 gigs, Prod server shows only 63 gigs of 128 utilized after testing multiple query's. Test server is hovering about 250 gigs but has a lot more going with other projects.
Really hoping someone can tell me I missed something stupid.
My next test is rebuilding the server with sql Dev version to see if it is something with sql standard. My last straw is trying to convince the sys admin and management to let me run a linux server in production.
Ryan
(63 rep)
Jul 25, 2024, 09:53 PM
• Last activity: Jul 26, 2024, 08:25 PM
10
votes
6
answers
11911
views
Parallelism Best Practices
What are the best practices with setting parallelism in general? I know that SQL Server defaults to `0` to use all available processors, but in what instance would you want to change this default behavior? I remember reading somewhere (I'll have to look for this article) that for OLTP workloads you...
What are the best practices with setting parallelism in general? I know that SQL Server defaults to
0
to use all available processors, but in what instance would you want to change this default behavior?
I remember reading somewhere (I'll have to look for this article) that for OLTP workloads you should turn off parallelism (set maxdop to 1
). I don't think I completely understand **why** you would do this.
When would you keep maxdop up to SQL Server (0)?
When would you turn off parallelism (1)?
When would you explicitly state the maxdop to a particular number of processors?
What causes parallelism?
anon
Sep 29, 2011, 01:13 AM
• Last activity: Jul 24, 2024, 04:40 PM
1
votes
1
answers
1998
views
PostgreSQL: running SQL statements in parallel in a SQL script
I use a for loop in a script to compute geometries according to values in a series, and insert the results into a table (t1). Is there a way to rewrite the loop below using parallelism? What approach to use? I found many docs on how PostgreSQL runs queries in parallel, but hardy anything on how I co...
I use a for loop in a script to compute geometries according to values in a series, and insert the results into a table (t1). Is there a way to rewrite the loop below using parallelism? What approach to use? I found many docs on how PostgreSQL runs queries in parallel, but hardy anything on how I could perform such task in my case. Is it even possible using SQL? or should I rely on other languages like Python, and execute the SQL queries from there?
DO $$
DECLARE
lims int[];
lim int;
BEGIN
SELECT ARRAY(SELECT generate_series(60, 300, 60)) INTO lims;
CREATE TABLE t1(
lim integer,
the_geom geometry(polygon, 4326),
);
FOREACH lim IN ARRAY lims
LOOP
WITH conv_hull AS (
SELECT ST_ConcaveHull(...I use lim here...) AS conv_hull
)
INSERT INTO t1 (lim, the_geom) SELECT lim, ST_SetSRID(conv_hull.ST_ConcaveHull, 4326) FROM conv_hull;
END LOOP;
END $$;
Marc
(13 rep)
Oct 27, 2021, 06:36 PM
• Last activity: Jul 1, 2024, 09:05 AM
2
votes
1
answers
361
views
How should MAXDOP be configured on stacked instances?
There is good general advice for how to configure Maximum Degree of Parallelism (MAXDOP). However, I am in a special circumstance. There are two instances on my box. If I have 16 cores, how should I divide MAXDOP between them? What metrics would reveal that I have made a bad choice? I am on SQL Serv...
There is good general advice for how to configure Maximum Degree of Parallelism (MAXDOP). However, I am in a special circumstance. There are two instances on my box. If I have 16 cores, how should I divide MAXDOP between them? What metrics would reveal that I have made a bad choice?
I am on SQL Server 2019 and [the standard advice for MAXDOP changed in 2016](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/configure-the-max-degree-of-parallelism-server-configuration-option?view=sql-server-ver16#recommendations) so info from before June 2016 (the release date of SQL Server 2016) may not be applicable.
J. Mini
(1225 rep)
Jun 13, 2024, 06:16 PM
• Last activity: Jun 14, 2024, 02:37 AM
Showing page 1 of 20 total questions