Environment Variables and MS Machine Learning Services
0
votes
0
answers
29
views
My apologies in advance if this is a naive or obvious question, but I am relatively new to this sort of thing. Here is my basic set up and dilemma:
1. I have an ETL process that starts with an API call returning XML. The XML needs some extensive processing that is difficult to perform (at least for me as it involves quite a bit of REGEX) in SQL, but easy to perform with Python. The Python script writes the result to a SQL Server table.
2. The ETL process needs to be run on a schedule, but my organization wants to keep all scheduled server tasks in one job agent (MS SQL Server Job Agent).
3. MS Machine Learning Services lets one execute Python script on SQL Server, so this seems like the best solution.
Dilemma: the Python script needs a server username, password, and a private API key. I do not want to nor should I hardcode these values in, so storing them as environment variables seemed like the obvious solution. I defined these as system variables on the server and tried to access them with MS ML Services to no avail.
I am not sure about this, but it seems that ML services creates App Containers for running python script and these containers do not inherit system environment variables.
Am I missing something obvious? Thanks in advance for any suggestions/solutions!
Asked by Charlie
(1 rep)
Jan 16, 2024, 04:24 PM