How can we manage cross-database dependencies across environments?
5
votes
1
answer
1218
views
I've put off asking this question for some time since it's tricky to
encapsulate our situation and challenges without a wall of text, but the
situation is getting worse so I'll do my best. I'm looking for some assistance in improving the way we develop and manage our application database and developer environments, particularly in regards to database dependencies across environments.
### About us
We're a medium-sized company with a fair amount of legacy code. To give an sense of what our current application database looks like, here are some ballpark numbers: 50gb, 450 tables, 200 views and 400 stored procedures. In addition, our production server runs approximately 15 databases, most of which needs or is needed by our application database.
To clarify: When I say "need", I'm referring to database objects that will not
compile/will compile but not run without the dependency. Examples of these
objects are server objects like Linked Servers and Replication Subscriptions, or
database objects like Stored Procedures and Views.
Over the past year, we've made significant improvements to the way we develop
and deploy our database. Improvements so far include the introduction of
dedicated developer environments, version control of (nearly) all database code,
automated deployment from Git (trigger-based) and the transition to a SQL Server
cluster.
### The issue
What we're struggling with, and which I cannot seem to find suitable resources for, is how to deal with dependencies from our application database to other databases. These dependencies fall into two separate challenges:
**1. Databases on the same server**
At the moment of speaking, our application database depends upon 5 databases on the same server. These are databases with separate repositories, deployment pipelines, libraries and web projects. While bootstrapping developer environments, we must take care to create these in a particular order in order to successfully apply DDL and DML-scripts, lest we face dependency errors. This process alone is causing a lot of headache. In fact, it is causing so much headache that some of our developers have simply given up on local developer environments and do all of their development in a shared database.
**2. Databases on a remote server only accessible to production**
In our production environment, we import data from a handful of remote SQL Server instances. Some of this data is imported using stored procedures that reference remote servers using Linked Server objects. In order for the stored procedure to run, the Linked Server object needs to exist. In order for the Linked Server object to "successfully" exist, the remote server it references must be reachable. The remote servers are only accessible from our production server (and rightly so), but this causes our stored procedures not compile properly during deployment.
In the book "Continuous Delivery", author Dave Farley emphasizes that in true Continuous Integration, every single resource required to assemble and run a project should reside in its repository. Additionally, he specifies that every environment should be identical (with the exception of configurations such as credentials and connection strings). Our application does not satisfy these principles, and I'm not even sure if it's feasible to do so.
**Our tools**
- Database server: Microsoft SQL Server 2017 - Build server: Visual Team Services - Build tools: Redgate DLM Automation Suite It feels like I'm missing out on some core architectural principles here. What can we do to alleviate these issues? Suggestions to relevant literature are also welcome.
- Database server: Microsoft SQL Server 2017 - Build server: Visual Team Services - Build tools: Redgate DLM Automation Suite It feels like I'm missing out on some core architectural principles here. What can we do to alleviate these issues? Suggestions to relevant literature are also welcome.
Asked by krystah
(747 rep)
Jun 13, 2018, 01:28 PM
Last activity: Jun 23, 2018, 12:48 PM
Last activity: Jun 23, 2018, 12:48 PM