I am working on a project that has the potential to have a large number of users each of which will be managing their own unique data sets. I am thinking the data can be stored in one of two ways.
1) Create a completely different database for each user so that their data is fully separate from everyone elses
2) Share the data in the same database, and segregate it at the query level using a user_id field.
The schema will always be identical for each user.
The main thing is that the system will need to be able to scale, and I am not sure if having potentially several thousand different databases, or storing millions of records in the same tables would scale better.
I am interested in hearing from anyone who has dealt with this kind of situation in the past and what pitfalls might be out there with either option.
Option 2 should be your best bet. Databases are usually designed to work with millions and millions of rows and a lots of data. So, as long as you design your schema correctly and have proper indexes, fill factors etc., option 2 will lead you to the scaling that you are looking for. As DarthVader said, learn more about database design.