Home/ Questions/Q 7598527
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-30T22:25:42+00:00

I am writing a large number of SQL scripts to be shared by a

  • 0

I am writing a large number of SQL scripts to be shared by a number of people, who will generally work with the scripts by editing them slightly for their specific purpose, then using the modified script to generate a dataset for further analysis.

I’ve struggled with the best way to build and organize such a library. I want more structure than just a folder with a long list of files like “signups_and_spend_by_week.sql.” I see basically two, inter-related issues:

  • Parameterization. Things like dates and sample sizes are relatively easy to pull out and make into variables, but what about parameters that alter the nature of the query—say changing a left join to an inner join? Is it wise to just (a) comment these kinds of possibilites, (b) create two versions (hence creating serious DRY problems), or maybe (c) wrap queries with a higher-lever programming language that can more easily represent these kinds of possibilites, e.g.,

       q = "SELECT * FROM plants" 
   if want_all: 
       q = _q + "LEFT JOIN fruits ON fruits.plant_id = plants.id"
   else: 
       q = _q + "INNER JOIN fruit ON fruits.plant_id = plants.id"       

  run_query(q)

  • DRY. I find that the same patterns continually appear, with similar or identical views being created as intermediate steps for more complex queries. My thinking is to split these re-usable bits out and call them as needed. However, my concern is that this then creates dependencies all over the library, and that seemingly innocuous changes to a “base” query might might inadvertently break someone else’s query. Other than just discipline and good documentation/rules, is their some compromise possible, e.g., automated testing?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    This looks like a problem that should not be treated in a purely technical way.

    Basically you will offer a base of SQL’s that other people can refine.
    Some of your base SQL’s will have bugs, run slow in certain circumstances, solve problems that don’t exist, are spot on, … and they will be improved, altered, neglected, praised, …

    It is hard to foresee what will work best for you and your co-workers. At least I don’t have an idea.

    I would suggest that you first start very basic, very simple: with a directory with SQL scripts, giving the files meaningful names.
    – Allow people to look at them, use them, change them, derive from them, comment on them, give points for their usefulness, ….
    – Have frequent meetings with all involved.
    – Try to find out what works well, what your group needs, what things don’t work at all, ….
    – Only when you, as a group, start to see clearly what tools you need to support what you’re doing, start thinking and designing a system to supports your needs. Only then store your scripts in a database, if that still makes sense.

    Don’t start designing now, you will probably throw 80% away, and the 20% you keep will be to spare the feelings of the one who has spent so much time up front trying to come up with something useful for the group.

    For me this is really a situation where a SCRUM approach works best: a situation that no one has a clear view of on how it should be best build. Communication, short sprints that try to address problems, interactivity, change the things that didn’t work out as planned, … those seem to me the key words and phrases for this project.

    Let it grow, don’t assume you can grok now how it will be.

    (I wrote this out of the assumption that it is as clear to you as it is to me how your project will evolve. If you have a clear view of what it will look like and have done this before, my comments will be way off.)

    • 0