Database Administrators
Q&A for database professionals who wish to improve their database skills
Latest Questions
0
votes
1
answers
14
views
Is it possible to run Django migrations on a Cloud SQL replica without being the owner of the table?
I'm using **Google Cloud SQL for PostgreSQL** as an **external primary replica**, with data being replicated continuously from a self-managed PostgreSQL source using **Database Migration Service (DMS)** in CDC mode. I connected a Django project to this replica and tried to run a migration that renam...
I'm using **Google Cloud SQL for PostgreSQL** as an **external primary replica**, with data being replicated continuously from a self-managed PostgreSQL source using **Database Migration Service (DMS)** in CDC mode.
I connected a Django project to this replica and tried to run a migration that renames a column and adds a new one:
uv run python manage.py migrate
However, I get the following error:
django.db.utils.ProgrammingError: must be owner of table camera_manager_invoice
This makes sense, since in PostgreSQL, ALTER TABLE
requires table ownership. But in this case, the replica was created by DMS, so the actual table owner is the replication source — and not the current user.
---
## 🔍 The Problem:
I'm trying to apply schema changes via Django migrations on a Cloud SQL replica that I do **not own**. The replication is working fine for data (CDC), but I need to apply structural changes on the replica independently.
---
## ✅ What I Tried:
* Changing the connected user: still not the owner, so same error.
* Running sqlmigrate
to get the SQL and applying manually: same result — permission denied.
* Attempted to change ownership of the table via ALTER TABLE ... OWNER TO ...
: failed due to not being superuser.
* Tried running migration with --fake
, but this skips execution and doesn't change the schema.
---
## ❓ My Question:
> **Is there any way to apply schema changes via Django migrations (or manually) on a Cloud SQL replica, without being the table owner?**
I'm open to alternatives, best practices, or official GCP recommendations for this situation.
---
Raul Chiarella
(117 rep)
Jul 22, 2025, 06:40 PM
• Last activity: Jul 25, 2025, 02:53 PM
1
votes
1
answers
25
views
Why is my Cloud SQL external replica not reflecting schema changes (like new columns) after Django migrations?
I'm using **Google Cloud Database Migration Service (DMS)** to replicate data from a self-managed PostgreSQL database into a **Cloud SQL for PostgreSQL instance**, configured as an *external primary replica*. The migration job is running in **CDC mode** (Change Data Capture), using **continuous repl...
I'm using **Google Cloud Database Migration Service (DMS)** to replicate data from a self-managed PostgreSQL database into a **Cloud SQL for PostgreSQL instance**, configured as an *external primary replica*.
The migration job is running in **CDC mode** (Change Data Capture), using **continuous replication**. Everything seems fine for data: new rows and updates are being replicated successfully.
However, after running Django’s
makemigrations
and migrate
on the source database — which added new columns and renamed others — **the schema changes are not reflected in the Cloud SQL replica**. The new columns simply don’t exist in the destination.
### 🔍 What I’ve done:
- Source: self-managed PostgreSQL instance.
- Target: Cloud SQL for PostgreSQL set as an external replica.
- Replication user has proper privileges and is connected via mTLS.
- The job is active, with "Optimal" parallelism and healthy status.
- Data replication (INSERT/UPDATE/DELETE) works great.
- Schema changes like ALTER TABLE
, ADD COLUMN
, RENAME COLUMN
are **not reflected** in the replica.
---
### ❓ Question:
**How can I configure DMS or Cloud SQL to also replicate schema changes (like ALTER TABLE or CREATE COLUMN) from the source to the replica? Or is it necessary to manually apply schema changes on the target?**
> I'm fine with workarounds or official recommendations — just need clarity on the correct approach for schema evolution in this setup.
---
Raul Chiarella
(117 rep)
Jul 22, 2025, 06:05 PM
• Last activity: Jul 25, 2025, 02:48 PM
3
votes
2
answers
662
views
Getting Postgres CREATE TABLE statements
I created some tables (9, to be exact) with a Django (1.9.6) migration and now I'm trying to get simple `CREATE TABLE` statements for them. I tried [this answer](https://stackoverflow.com/a/2594564/3704831), but using `pg_dump` in this way gives me over 800 lines of output for the 9 tables. For exam...
I created some tables (9, to be exact) with a Django (1.9.6) migration and now I'm trying to get simple
CREATE TABLE
statements for them. I tried [this answer](https://stackoverflow.com/a/2594564/3704831) , but using pg_dump
in this way gives me over 800 lines of output for the 9 tables. For example, part of the output creating the first table is
--
-- Name: popresearch_question; Type: TABLE; Schema: public; Owner: postgres
--
CREATE TABLE popresearch_question (
id integer NOT NULL,
created_date timestamp with time zone NOT NULL,
modified_date timestamp with time zone NOT NULL,
question_text character varying(500) NOT NULL,
question_template_id integer,
question_type_id integer,
user_id integer
);
ALTER TABLE popresearch_question OWNER TO postgres;
--
-- Name: popresearch_question_id_seq; Type: SEQUENCE; Schema: public; Owner: postgres
--
CREATE SEQUENCE popresearch_question_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE popresearch_question_id_seq OWNER TO postgres;
--
-- Name: popresearch_question_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: postgres
--
ALTER SEQUENCE popresearch_question_id_seq OWNED BY popresearch_question.id;
and then later on are more ALTER
statements:
--
-- Name: popresearch_question id; Type: DEFAULT; Schema: public; Owner: postgres
--
ALTER TABLE ONLY popresearch_question ALTER COLUMN id SET DEFAULT nextval('popresearch_question_id_seq'::regclass);
and then later:
--
-- Name: popresearch_question popresearch_question_pkey; Type: CONSTRAINT; Schema: public; Owner: postgres
--
ALTER TABLE ONLY popresearch_question
ADD CONSTRAINT popresearch_question_pkey PRIMARY KEY (id);
--
-- Name: popresearch_question popresearch_question_question_text_key; Type: CONSTRAINT; Schema: public; Owner: postgres
--
ALTER TABLE ONLY popresearch_question
ADD CONSTRAINT popresearch_question_question_text_key UNIQUE (question_text);
and after that there are at least a dozen more ALTER TABLE
statements just for this one table scattered in the pg_dump
output. Is there a way to get a simple, condensed CREATE TABLE
statement that includes all the keys, constraints, etc.?
wogsland
(416 rep)
May 9, 2017, 09:47 PM
• Last activity: Jul 3, 2025, 07:00 PM
0
votes
1
answers
720
views
Django Multi Table Inheritance and Preserving Child and Child History
Place can have many different one to one fields, that’s why it can have bookstore, restaurant, and hardware store. But can the same place be both a bookstore and a restaurant at the same time? Now if the bank and restaurant tables don't have the same pk as place, I would think the answer is yes. But...
Place can have many different one to one fields, that’s why it can have bookstore, restaurant, and hardware store. But can the same place be both a bookstore and a restaurant at the same time? Now if the bank and restaurant tables don't have the same pk as place, I would think the answer is yes. But I also know that unless you put parent_link=True on it, erasing the child row automatically deletes the parent row.
My use case is to preserve the history of Restaurant despite it’s having now become a Bookstore, and if a Restaurant is simultaneously a Bookstore, to be able to keep both places in my db. Is the best way to do this with a fk to each of the bookstores, restaurants and hardware stores instead of either a OneToOne or multi table inheritance? Or is there some other way I’m not aware of? This has to be a solved problem, but so far I haven't found it. I'm currently looking at NFL databases because players can be on more than one team, (albeit not at the same time), and they have a recorded history of their team and individual stats - to see if I can hack those into what I want.
I'm even willing to consider an ArrayField or Hstore. I'm on Postgres 9.4 and trying to maintain 3NF. All wisdom accepted. Thanks.
Malik A. Rumi
(377 rep)
Aug 24, 2016, 01:54 AM
• Last activity: May 27, 2025, 02:09 PM
0
votes
2
answers
78
views
Aggregate the last two entries per item efficiently
I'd like your help to optimise an increasingly slow query - or better yet, help me understand what is the problem and point me in the right direction. Every day I scrape 3 supermarkets and I record their prices in a django app. Each scrape results in a new `Retrieval` record, a few thousand `Price`...
I'd like your help to optimise an increasingly slow query - or better yet, help me understand what is the problem and point me in the right direction.
Every day I scrape 3 supermarkets and I record their prices in a django app. Each scrape results in a new
Retrieval
record, a few thousand Price
records and maybe in a few Item
records (most items already exist, so I only record their prices for a given day).
Once this is done, a process runs which computes the price changes for all of today's items, by retrieving each item's **last two** prices. These last two prices may not always be on consecutive days, because some items come and go.
The 3 tables that we're interested in and their indexes look like this:
CREATE TABLE public.main_item (
id uuid NOT NULL,
name character varying(512) NOT NULL
);
CREATE TABLE public.main_price (
id uuid NOT NULL,
per_item double precision NOT NULL,
item_id uuid NOT NULL,
retrieval_id uuid NOT NULL
);
CREATE TABLE public.main_retrieval (
id uuid NOT NULL,
"timestamp" timestamp with time zone NOT NULL,
);
CREATE INDEX name_index ON public.main_item USING btree (name);
CREATE INDEX timestamp_index ON public.main_retrieval USING btree ("timestamp");
I have a query which returns a result that looks like this where id
is the Item
id and latest_prices
is a json which contains the last 2 prices for that item and then I process the results in python.
| id | latest_prices|
|-|-|
0003db22-3c8a-4f21-aea1-667361ebe377 | {"{\"per_item\": 2.44, \"price_id\": \"24dc5524-35d5-472b-8f16-5840308a9cc4\"}","{\"per_item\": 2.44, \"price_id\": \"a415d740-0e50-43ba-b33f-3d6c9328a319\"}"}
0011cc73-07ca-415d-85e0-1c6782e0b041 | {"{\"per_item\": 3.48, \"price_id\": \"e754cc25-9fb5-4e88-8689-55878e47f7dc\"}","{\"per_item\": 3.48, \"price_id\": \"553a6cf2-2c6e-421c-b7e0-c43d5c0cbf85\"}"}
However, the query is getting increasingly slow. My assumption is because of the size of the Price
table which by now is ~16 million rows.
Running an EXPLAIN ANALYZE
on that query I see that the majority of the time is spent in a Bitmap Heap Scan
on the main_price
table: https://explain.depesz.com/s/ZX78#stats
Below you can find the query that the django ORM has generated for today:
SELECT "main_item"."id", ARRAY(
SELECT JSONB_BUILD_OBJECT(('price_id')::text, U0."id", ('per_item')::text, U0."per_item") AS "json"
FROM "main_price" U0
INNER JOIN "main_retrieval" U2
ON (U0."retrieval_id" = U2."id")
WHERE (U0."item_id" = ("main_item"."id") AND U2."timestamp" = '2024-09-23 00:00:00+00:00'
AND V0."timestamp" < '2024-09-24 00:00:00+00:00'
AND NOT (
EXISTS(
SELECT 1 AS "a"
FROM "main_retrieval" U0
LEFT OUTER JOIN "main_price" U1
ON (U0."id" = U1."retrieval_id")
WHERE (U1."item_id" IS NULL AND U0."id" = (V0."id"))
LIMIT 1))))
### Note
I'm more than happy to ditch the query generated by the ORM and write it by hand.
However, I'm struggling to find an optimal way to fetch the 2 most recent prices for each Item. Fetching the most recent is easy, but fetching the previous one is surprisingly difficult.
Sakis Vtdk
(103 rep)
Sep 27, 2024, 04:58 PM
• Last activity: Oct 2, 2024, 08:13 PM
1
votes
0
answers
608
views
Railway - problems with connecting to postgres in production
**Django worked perfectly locally, but had errors in production (I was using the same postgres DB and configuration).** **Django server, hosted on Railway gave me this:** [![enter image description here][1]][1] **Newtorking tab in Postgres settings on Railway:**[![enter image description here][2]][2...
**Django worked perfectly locally, but had errors in production (I was using the same postgres DB and configuration).**
**Django server, hosted on Railway gave me this:**
**Newtorking tab in Postgres settings on Railway:**
**Postgres logs locally:**
**Settings:** (WORKED PERFECTLY ON A LOCAL MACHINE WITH THIS IN-PROD DATABASE AND SAVED DATA)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2', //tried django.db.backends.postgresql, not working
'NAME': os.getenv('DB_NAME'),
'USER': os.getenv('DB_USER'),
'PASSWORD': os.getenv('DB_PASSWORD'),
'HOST': os.getenv('DB_HOST'),
'PORT': os.getenv('DB_PORT'),
}
}
DATABASE_URL = os.getenv('DATABASE_URL')
**.env:** (everything in {{}} is a variable)
DATABASE_PRIVATE_URL={{DATABASE_PRIVATE_URL}} //not using this
DATABASE_URL={{DATABASE_URL}}
DB_HOST=viaduct.proxy.rlwy.net
DB_NAME=railway
DB_PASSWORD={{DB_PASSWORD}}
DB_PORT=19232
DB_USER=postgres
SECRET_KEY={{SECRET_KEY}}
**So I did:**
@my-MBP django_project % psql -h localhost -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+-----------------+----------+---------+-------+-----------------------
postgres | | UTF8 | C | C |
template0 | | UTF8 | C | C | =c/ +
| | | | | =CTc/
template1 | | UTF8 | C | C | =c/ +
| | | | | =CTc/
(3 rows)
@my-MBP django_project % psql -h localhost -d postgres
psql (14.11 (Homebrew))
Type "help" for help.
postgres=#
**And got (On Railway):**
***Django server:***
***Postgres Database:***
**Doing this locally gives me the same error that was given in production**
@my-MBP django_project % psql -h /var/run/postgresql -d postgres
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
Is the server running locally and accepting connections on that socket?
Again, before I ran





psql -h localhost -d postgres
everything was working perfectly on a LOCAL Django server. But in production, with the same configuration and database (hosted) it was not working.
Mikheil
(11 rep)
May 8, 2024, 04:40 AM
• Last activity: May 8, 2024, 01:52 PM
0
votes
1
answers
1035
views
How to investigate intermittent Postgres connection errors?
Our webserver (Django-app on Gunicorn running on Google Cloud Run) connects to a Postgres 15 database (on Google Cloud SQL) through Psycopg. Most queries are successful, but recently ~1% of the queries fail on random moments, with random error messages like: - `connection failed: region:db_name/.s.P...
Our webserver (Django-app on Gunicorn running on Google Cloud Run) connects to a Postgres 15 database (on Google Cloud SQL) through Psycopg. Most queries are successful, but recently ~1% of the queries fail on random moments, with random error messages like:
- `connection failed: region:db_name/.s.PGSQL.5432" failed:
server closed the connection unexpectedly - This probably means the
server terminated abnormally before or while processing the request.`
-
got message type "P", length 1380524074
- `connection failed:
region:db_name/.s.PGSQL.5432" failed: FATAL: password
authentication failed for user "postgres"`
- `consuming input failed:
server closed the connection unexpectedly - This probably means the
server terminated abnormally before or while processing the request.`
- invalid socket
Sometimes we see an error at the server side at the same moment, for example:
- FATAL: canceling authentication due to timeout
- FATAL: connection to client lost
- FATAL: password authentication failed for user
The password authentication failed
error puzzles me: we're always connecting with the same password.
The got message type "P"
looks cryptic to me, and the length mentioned (over 1G!) is abnormal, I don't see why such a long message is being sent.
In the Django settings file, I tried different settings:
- CONN_HEALTH_CHECKS
= True
or False
- CONN_MAX_AGE
= 0
(new connection for every request) or None
(unlimited persistent connections)
Resources (CPU, memory, disk space, ...) are well below the limits.
We use Unix sockets to connect to the database. Config in Django settings.py
:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'HOST': '/cloudsql/project:region:instance',
'NAME': ...,
'USER': ....,
'PASSWORD': ...,
}
}
Does anyone have any ideas on how I can investigate this problem?
Koen De Wit
(111 rep)
Jan 22, 2024, 03:06 PM
• Last activity: Mar 22, 2024, 02:36 PM
0
votes
2
answers
42
views
Question about multi fields index in MySQL
I'm having an doubt about coumpound indexes in my application. My db engine is Mysql 8.0 with INNODB and my question is the following: My application has a model called Report with the follwing fields `date`, `currency`, `type`, `company`,`aggregator` and `daily_value`. My applications consumes even...
I'm having an doubt about coumpound indexes in my application. My db engine is Mysql 8.0 with INNODB and my question is the following:
My application has a model called Report with the follwing fields
date
, currency
, type
, company
,aggregator
and daily_value
. My applications consumes events from kafka where I receive a value
that represents a monetary transaction and the corresponding company
. I'm using django and I make the following query each time I receive an event:
SELECT company,currency,date,aggregator,type
WHERE company=passed_company,currency=passed_currency,
type=passed_type,aggregator=passed_aggregator,
date=RANGE(start_date,end_date)
to retrieve the given report for that day if it exists and increment the daily_value
property with the event value
or create the report for that day if it doesn't exist. Basically I do a get_or_create
in Django for the ones who are familiar with this framework.
My question is should I make a coumpound index on date, currency, type, company, aggregator
to optmimize my query and avoid possible deadlocks? I'm asking this because I will have multiple parallel consumers and I want to avoid concurrency issues.
Pedro Silva
(1 rep)
Jul 20, 2023, 09:52 AM
• Last activity: Jul 20, 2023, 04:49 PM
0
votes
0
answers
535
views
How to debug this tuple await events in RDS PostgreSQL
I have been noticing that sometimes we get a lot of open connections for a long time. Enabling enhanced metrics in RDS PostgreSQL I saw the following: [![enter image description here][1]][1] It seems this tuple wait event is making a lot of queries wait. I have taken a look and it seems it is from l...
I have been noticing that sometimes we get a lot of open connections for a long time. Enabling enhanced metrics in RDS PostgreSQL I saw the following:
It seems this tuple wait event is making a lot of queries wait. I have taken a look and it seems it is from locks. I have run the query they suggest in the [docs](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/wait-event.locktuple.html) but it does not return anything, so not sure what is happening. In the Top sql queries you can see the following:
The queries are something like this (sorry for it being incomplete, RDS does not give me the full query):
~~~sql
UPDATE "customers_customer" SET "follow_up_date" = (CASE WHEN ("customers_customer"."id" = 140005033) THEN NULL WHEN ("customers_customer"."id" = 138501530) THEN NULL WHEN ("customers_customer"."id" = 140000447) THEN NULL WHEN ("customers_customer"."id" = 139999963) THEN NULL WHEN ("customers_customer"."id" = 140016560) THEN NULL WHEN ("customers_customer"."id" = 139998510) THEN NULL WHEN ("customers_customer"."id" = 138485030) THEN NULL WHEN ("customers_customer"."id" = 138507549) THEN NULL WHE
~~~
I'm not sure but I think this query comes from a mass update call on the Django ORM to a the customer model on the follow up date field.
So how can I debug this problem? Or mitigate it in the future? For now is not killing the database, but queries were starting to pile up and we have had problems with that in the past. If you need more information, please drop a comment below!


Antonio Gamiz Delgado
(111 rep)
May 17, 2023, 03:29 PM
• Last activity: May 17, 2023, 04:16 PM
0
votes
2
answers
6898
views
Vehicle database design - Type, Make, Model, Vehicle
I kindly ask you for advice, if I am correct with my DB design. I am writing an app in Django (it has its own ORM), which will use PostgreSQL. My expectation is, that I will specify: - Type (example: Automobile, Motorcycle, Boat, ...) - Make (example: Honda, BMW, Audi, ...) - Model (example: CBR1000...
I kindly ask you for advice, if I am correct with my DB design. I am writing an app in Django (it has its own ORM), which will use PostgreSQL.
My expectation is, that I will specify:
- Type (example: Automobile, Motorcycle, Boat, ...)
- Make (example: Honda, BMW, Audi, ...)
- Model (example: CBR1000, A6, A8, 535i, ...)
- Vehicle - all other info, like VIN, price, ...
So I think that the best way is to have Makes unique. I don't know, what to do with a **Type**. I think that the best way is to have a foreign key of a Type in a Model - because for example **Honda** can produce **Automobiles** and also **Motorcycles**. So by **Model** we can found, what vehicle type is it.
I want to make a web form filter (some kind of three linked dropdowns), where the user will select a **Type**, then it will filter out all **Makes** for that selected type. Then in another dropdown, he will select a Make and in the third dropdown, there will be shown all Models, which is available in that selection.
Something like a Copart has - Vehicle Finder on the right side of web page https://www.copart.com/vehicleFinder/
I hope that it makes sense, I have created a schema like this, and I am not just sure if it is usable and it is right. Here it is:
link to editor: https://dbdesigner.page.link/gS62T3GYPth2TAAbA
Thank you very much for any advice!

saby
(105 rep)
Dec 2, 2020, 05:21 PM
• Last activity: Apr 29, 2023, 03:00 AM
1
votes
0
answers
85
views
Migrating PostgreSQL (Django REST Framework) to slightly different PostgreSQL (FastAPI). Best way to do it?
I need to migrate data from a relatively small PostgreSQL database (let's call it **DB 1**) of a Django REST Framework project to another PostgreSQL database of my FastAPI project (**DB 2**). Say 500,000 rows. The tables are slightly different. DB 1 uses *IDs as pk*, while the second uses *UUIDs*. T...
I need to migrate data from a relatively small PostgreSQL database (let's call it **DB 1**) of a Django REST Framework project to another PostgreSQL database of my FastAPI project (**DB 2**). Say 500,000 rows.
The tables are slightly different. DB 1 uses *IDs as pk*, while the second uses *UUIDs*. There are fk linking the tables.
So, I have to move some

Users
after changing the ID
to UUID
, and the name
to last/first name
.
The DBs are on different servers.
Both databases will be running in parallel for a few months, but traffic will be low. And finally, I will be moving some users initially and later on the rest.
---
I was thinking of adding columns in DB1 (which is in production) and expose an endpoint for the transfer. My 2nd project will then call the endpoint in my 1st and request the transformed data.
Added columns:
- UUIDs,
- Last Name, First Name,
- "Already transferred to DB2"
The new columns in DB1 will only be used for the migration. Nothing else.
The benefits are:
- Single(ish) source of truth
- less bug prone since I'll be using Django (models, Fields, etc.)
Cons:
- I am editing production code and production DB
**Is that the right way to do it?** Is there a better way?
---
**Note:** _I have already implemented it as an external script that connects directly to the DBs (SQLAlchemy) and transforms the data (Pandas), but I fear it will become too complex to handle eventually._
PythonForEver
(133 rep)
Apr 3, 2023, 07:50 AM
• Last activity: Apr 3, 2023, 02:45 PM
0
votes
1
answers
1572
views
How to upgrade Mysql 5.5 to 5.6 without data loss on aws-ec2?
MySQL server (5.5) is running on aws-ec2 instance with following linux details ``` NAME="Amazon Linux AMI" VERSION="2015.09" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2015.09" PRETTY_NAME="Amazon Linux AMI 2015.09" ANSI_COLOR="0;33" CPE_NAME="cpe:/o:amazon:linux:2015.09:ga" HOME_URL="http://aws.am...
MySQL server (5.5) is running on aws-ec2 instance with following linux details
NAME="Amazon Linux AMI"
VERSION="2015.09"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.09"
PRETTY_NAME="Amazon Linux AMI 2015.09"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.09:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/ "
Since Django 2.2 doesn't support Mysql 5.5
. I want to upgrade MSQL to 5.6 .
- We don't have backup server and we don't want new server .
- Creating sql dump , removing MySQL 5.5 then installing MySQL 5.6 and importing dump will it work ? Since server has 100 GB of data importing the dump will be a long process and we will have considerable amount of downtime .There can be some issue with data while importing dump in 5.6 .
Is there a way to upgrade MySQL without data loss and less downtime ?
Palash Jadhav
(9 rep)
Jul 21, 2020, 04:37 PM
• Last activity: Apr 8, 2022, 02:04 AM
1
votes
2
answers
15164
views
How to fix all duplicate key values from sequences?
I have a Postgres database that I created with a Django application. When I transferred it over, it never reindexed anything and it gives me errors trying to do this. It seems as if there are lots of tables that have this issue. Is there a way I can use this code that I found on another question and...
I have a Postgres database that I created with a Django application. When I transferred it over, it never reindexed anything and it gives me errors trying to do this. It seems as if there are lots of tables that have this issue. Is there a way I can use this code that I found on another question and use it for all sequences and tables?
SELECT setval('tablename_id_seq', (SELECT MAX(id) FROM tablename)+1)
Lewis Menelaws
(111 rep)
Dec 19, 2017, 09:52 PM
• Last activity: Mar 6, 2022, 01:13 AM
0
votes
1
answers
1730
views
Create table with foreign keys blocked by other Table-level Lock
I'm not pretty sure it is database question or development-process question, but I will kindly give it a try. When creating a new table B with foreign keys to table A, the operation tries to acquires SHARE ROW EXCLUSIVE lock on the referenced table (table A). However if table A is being busy by vari...
I'm not pretty sure it is database question or development-process question, but I will kindly give it a try.
When creating a new table B with foreign keys to table A, the operation tries to acquires SHARE ROW EXCLUSIVE lock on the referenced table (table A). However if table A is being busy by various UPDATES (ROW EXCLUSIVE locks). This cause my database migration locks for too long time (15+ minutes) and risk my production. In Some occasions I manually assist my db migration by killing the workers directed to table A.
Beside scheduling almost *every* db migration to occurs in the off-hours (1AM-5AM), can you suggest a better or creative solution to help my team deploy "*naive*" PR to production.
References:
1. https://www.postgresql.org/docs/12/sql-createtable.html
2. https://www.postgresql.org/message-id/75218696-61be-4730-89f6-dd6058fa9eda@a28g2000prb.googlegroups.com
Cowabunga
(145 rep)
Nov 5, 2021, 01:59 AM
• Last activity: Nov 5, 2021, 05:05 AM
2
votes
1
answers
2333
views
Sensible max_length for Postgres db charfields
I am using Django with a Postgres backend. In the Django models, I have to set the max_length attribute on the Charfields, then the Django ORM deals with the database in the backend. In a lot of cases, those are rather small fields (e.g. say names, or city names etc.) that I don't expect would be ve...
I am using Django with a Postgres backend.
In the Django models, I have to set the max_length attribute on the Charfields, then the Django ORM deals with the database in the backend. In a lot of cases, those are rather small fields (e.g. say names, or city names etc.) that I don't expect would be very long.
That application deals with existing data that isn't particularly cleaned. So I don't want to set something like
name = models.Charfield(max_length=50, ...)
Just to realize that someone triple wrote his name in that field and that the field is 75 characters long. On the other hand, I don't want to set huge values and needlessly increase database size just to save me a little bit of trouble.
My current thinking is that Postgres probably reserves some minimal amount of space (e.g. likely that max_length=10 and max_length=15 both take the same storage), so I would set the max_length to that minimal size (or low multiple thereof).
What do you recommend and why?
logicOnAbstractions
(143 rep)
Nov 1, 2021, 05:50 PM
• Last activity: Nov 2, 2021, 06:37 AM
6
votes
2
answers
11311
views
Django postgres multiple schema
I'm trying to develop a django interface for an existing postgresql db, the db makes use of various schemas, looking at the literature the following example should work, but it only returns the schema defined in the default database when I run `python manaage.py inspectdb`. Also when this works, how...
I'm trying to develop a django interface for an existing postgresql db, the db makes use of various schemas, looking at the literature the following example should work, but it only returns the schema defined in the default database when I run
python manaage.py inspectdb
. Also when this works, how do I define which schema to use when defining the django model?
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'OPTIONS' : {
'options': '-c search_path=public'
},
'NAME': 'gygaia',
'USER':'postgres',
'PASSWORD':'abc',
'HOST':'localhost',
'PORT':'5432',
},
'samples': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'OPTIONS' : {
'options': '-c search_path=samples'
},
'NAME': 'gygaia',
'USER':'postgres',
'PASSWORD':'abc',
'HOST':'localhost',
'PORT':'5432',
},
'excavation': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'OPTIONS' : {
'options': '-c search_path=excavation'
},
'NAME': 'gygaia',
'USER':'postgres',
'PASSWORD':'abc',
'HOST':'localhost',
'PORT':'5432',
},
}
Spatial Digger
(207 rep)
Jun 5, 2018, 07:17 PM
• Last activity: Oct 31, 2021, 05:39 AM
1
votes
0
answers
286
views
Storing user defined forms and their data
I'm writing an application that would have a feature similar to Google Forms. In this feature, some end users of the system will have the ability to design custom forms. Those forms will subsequently be filled by other users. So there are two classes of data that need to be stored in the database: -...
I'm writing an application that would have a feature similar to Google Forms.
In this feature, some end users of the system will have the ability to design custom forms. Those forms will subsequently be filled by other users. So there are two classes of data that need to be stored in the database:
- Form definition: form sections, fields, types of fields, choices available for certain fields, whether the user is confined to selecting one or allowed multiple choices, whether the field is optional or required, etc.
- Form data: an instance of a filled form. It has a many-to-one relationship with form definition.
The system needs to allow users of the system to collaboratively design their own forms and modify them over time. To keep things sane, a form can only be used once it is "committed" or published, after which time the form schema cannot be modified.
Users can also fill instances of these forms. They can save a partially filled instance that hasn't been completely validated yet, as a draft. Once they "file" the form, it gets validated and if valid it will not be modifiable anymore, but will be stored and becomes "view only". We will also need the ability to generate reports of responses to those forms and export form data as a CSV file.
My plan is to use JSONB fields in my PostgreSQL database to store both the form definition as well as the form filing.
Is there a way to use the form definition JSON blob to automatically derive some sort of schema against which form filing instances can be constrained?
Are there open source examples or academic papers discussing this sort of design which we can learn or borrow from?
hashemi
(111 rep)
Oct 7, 2021, 04:40 AM
0
votes
1
answers
1331
views
Just delete 50.000.000 of rows on a PostgreSQL table and DB still very slow
sorry for my poor english. I have a postgres DB runing on amazon RDS (db.t3.small), with django as a backend. i have made a mistake and created 50.000.000 rows. when i figure out (because queries on that table where ultra slow) i delete it all. but the queries i make on that table stills super slow....
sorry for my poor english.
I have a postgres DB runing on amazon RDS (db.t3.small), with django as a backend. i have made a mistake and created 50.000.000 rows. when i figure out (because queries on that table where ultra slow) i delete it all. but the queries i make on that table stills super slow. it only have 300 rows now.
i have to clean some cache? i have to wait something? the configuration of the RDS in aws is default.
the engine version of postgres is 12.5, also have postgis installed in it.
i check for vacuum issues and run this command:
SELECT relname AS TableName,n_live_tup AS LiveTuples,n_dead_tup AS DeadTuples,last_autovacuum AS Autovacuum,last_autoanalyze AS Autoanalyze FROM pg_stat_user_tables;
the table with the problem says:
'appointment_timeslot', 1417, 0, datetime.datetime(2021, 7, 21, 18, 13, 8, 193967, tzinfo=), datetime.datetime(2021, 7, 21, 18, 13, 30, 551750, tzinfo=)
check for indexes that Django creates automaticly on that table and i find 4
[
('appointment_timeslot_pkey', 'CREATE UNIQUE INDEX appointment_timeslot_pkey ON public.appointment_timeslot USING btree (id)')
'appointment_timeslot_home_visit_id_62df4faf', 'CREATE INDEX appointment_timeslot_home_visit_id_62df4faf ON public.appointment_timeslot USING btree (home_visit_id)')
('appointment_timeslot_office_id_6871b47b', 'CREATE INDEX appointment_timeslot_office_id_6871b47b ON public.appointment_timeslot USING btree (office_id)')
('appointment_timeslot_time_range_id_715578fa', 'CREATE INDEX appointment_timeslot_time_range_id_715578fa ON public.appointment_timeslot USING btree (time_range_id)')
]
Nahuel Fernandez
(3 rep)
Jul 22, 2021, 12:39 AM
• Last activity: Jul 22, 2021, 04:44 AM
0
votes
0
answers
30
views
Reduce mysql query ( Fulltext-search ) execution runtime, server-side?
****VPS**:** - ubuntu - Django (latest) - Python3.6 - Mysql **Structure :** CREATE TABLE `tablename` ( `id` int(11) NOT NULL AUTO_INCREMENT, `username` varchar(255) CHARACTER SET latin1 NOT NULL, `wallpaper` varchar(200) CHARACTER SET latin1 NOT NULL, `timestamp` datetime(6) NOT NULL, `tags` text CH...
****VPS**:**
- ubuntu
- Django (latest)
- Python3.6
- Mysql
**Structure :**
CREATE TABLE
tablename
(
id
int(11) NOT NULL AUTO_INCREMENT,
username
varchar(255) CHARACTER SET latin1 NOT NULL,
wallpaper
varchar(200) CHARACTER SET latin1 NOT NULL,
timestamp
datetime(6) NOT NULL,
tags
text CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
downloads
int(11) NOT NULL,
wallpaper_preview
varchar(200) CHARACTER SET latin1 NOT NULL,
views
int(11) NOT NULL,
wall_height
int(11) NOT NULL,
wall_size
varchar(10) NOT NULL,
wall_width
int(11) NOT NULL,
PRIMARY KEY (id
),
FULLTEXT KEY tags
(tags
)
) ENGINE=InnoDB AUTO_INCREMENT=14566 DEFAULT CHARSET=utf8
**Code ( from django views.py ):**
posts = mytable.objects.raw('SELECT id,
MATCH (tags) AGAINST (\''+some.tags+'\') as score
FROM table ORDER BY score desc;')
**Excution Time (5 runs in sec, views.py):**
6.103515625e-05
6.4849853515625e-05
6.318092346191406e-05
8.034706115722656e-05
8.273124694824219e-05
**Runtime checked using:**
start = time.time()
posts = modelname.objects.raw('SELECT id,
MATCH (tags) AGAINST (\''+some.tags+'\') as score
FROM table ORDER BY score desc;')
end = time.time()
print(f"Runtime views is {end - start}")
**Tried raw from ssh, mysql command line**
SELECT id, MATCH (tags) AGAINST ('car') as score
FROM tablename ORDER BY score desc;
**result:**
11834 rows in set (0.09 sec)
**Indexed using:**
ALTER TABLE 'tablename' ADD FULLTEXT('tags');
is there a better way. or something with Django ORM.
**example search:**
*'superscars'* searched in *'road,hometown,supercar,extremecars'*
if mysql is performing in 0.09 sec then why django takes so much time
, or i have done something wrong or missed something.
*i have asked this on stackoverflow, they redirected me here*
Harsh Jadon
(101 rep)
Jul 8, 2021, 09:37 PM
• Last activity: Jul 14, 2021, 06:42 PM
2
votes
2
answers
3958
views
Designing a multi-tenant database for a scenario with multiple user types
I'm developing a SaaS application for recruiting which has multiple user types such as `Client`, `Recruiter`, `Panellist` and `Owner` etc. There is also a chance that the user types may increase. The `Owner` has an `Organization` and this is what will be my tenant. Here I'm considering every owner's...
I'm developing a SaaS application for recruiting which has multiple user types such as
**NOTE**: I did not make the schema connections for the tables in the second tenant because it was starting to get messy but for all intents and purposes assume that they are present over there.
My questions are:
- What is exactly the best practice when dealing with multiple users in a multi-tenant scenario?
- What are the deficiencies of my design?
Client
, Recruiter
, Panellist
and Owner
etc. There is also a chance that the user types may increase.
The Owner
has an Organization
and this is what will be my tenant. Here I'm considering every owner's organization as a tenant. I'm using Django to develop this application and will be utilizing the schemas feature provided in the PostgreSQL, hence there will be a **schema** for every tenant.
What I'm planning to do is that there will be a global User
table that will be kept in the **public** schema and stores generic user information such a first name, last name, email, password, etc. The User
table will be used for logging into the site. And there will be additional tables for each of the respective user types that each have their own unique set of columns. These additional tables will be tenant/organization specific.
Now once the user logs into to site, he can then switch to his desired tenant/organization something similar to how workspaces function in Slack. Within the tenant/organization, the user can assume the role of the above-mentioned user types that were allotted by the Owner
to the user. So the user can be a Recruiter
in one tenant and he can also be a Panellist
in another tenant.
There will be an UserOrganization
junction table which will keep track of the fact that a User
can belong to multiple Organization
.
Here is a schema diagram of what I'm thinking:

Shahbaj Sayyad
(21 rep)
Apr 8, 2020, 10:08 AM
• Last activity: May 13, 2021, 04:58 PM
Showing page 1 of 20 total questions