Question

I’m just wondering what knowledge or techniques are needed to make a web framework like django.

so the webframework is able to serve as a cloud computing (a website can be scaled horizontally by sending some stuffs that must be solved to other server.) when needed, and can be used to build a website fast like django if a developer want to build a just simple website.

Sorry. my English is very awkward cause I’m an Korean.

just give me some approaches or instructions about what techniques are needed to build a web framework or what I have to do or learn.

Thanks a lot.

Answer 1

You’re asking a few questions here:

How to Make a Web Framework like Django

First, check out this article, How to use Python on the Web. It provides very good ideas on the various pieces that go into a framework.

Next, you need to ask yourself how much you want to create yourself.

From Scratch

If you’re doing this as an exercise in learning the ins and outs of frameworks, you should read up on WSGI. WSGI is the API that Django (and most other python frameworks) use to interface with the web server that actually handles the requests (like Apache, nginx and lighthttpd). It’s not necessarily a library (although there are libraries that help creating WSGI applications), but more of a standard API that, when used, promises that your code will work with other servers.

It looks like this:

def application(environ, start_response):
    # your implementation here

That’s it. environ is a dictionary that contains all of the CGI environment variables. start_response is a callback function that you call with the response headers and status code. Finally, the function is expected to return an iterator which is the response body. You can find out much more about this at the WSGI site.

With a “Wrapper” Library

If you don’t want to deal with the details of WSGI (which does have a few pitfalls that can cause some headaches), there are a few different libraries that wrap up the WSGI interface into something that’s slightly more user-friendly. For example, rather than having the repsponse headers be a list of string-tuples:

response_headers = [('Content-Type', 'text/plain'),
  ('Content-Length', str(len(response_body)))]

…they might allow you to have a response object that can set the headers through a dictionary…

response.headers['Content-Type'] = 'text/plain'
response.headers['Content-Length'] = len(response_body))

Django has it’s own request and response API, but there are others out there as well to check out. Off the top of my head, werkzeug, and WebOb.

Use an Existing Framework (Recommended)

However, if your goal is really just to create a website, and think that Django can’t do what you want it to (which probably isn’t the case, but I digress), then you don’t want to make your own framework at all. There are plenty out there that are more lightweight than Django, and allow you a bit more freedom to pick and choose your own libraries to work with (“modular”). This means they are designed with the idea that you might be using them for any purpose, rather than a framework like Django, which tries to steer you in the direction of using all of their tools. Check out frameworks like Flask, Bottle, and CherryPy.

Realize that all of these frameworks, libraries and APIs are just means to an end. Anything you want to do can probably be done in any of them, so it’s just a matter of finding the one that gives you the right combination of built-in tools that do all the low-level code that you don’t have to implement without adding to much high-level stuff from getting in your way.

Allow the Web Framework to be Served in the Cloud so that it can be Scaled Horizontally

This really isn’t a requirement of the web framework itself. Even using Django in the typical way (using their ORM and on a SQL database) can scale horizontally to a good extent (see the “Scaling” section in the Deployment chapter of the Django Book for some overview of how this is done; the ideas expressed there can actually be used for most horizontal scaling web applications). Reasonable horizontal scalability will be achieved by actually writing your application in a way that would allow it to scale horizontally. This typically means writing your server code in a way that assumes that it might handle requests on different physical servers. So, for example, if you store cookie data in a flat file on the web server, you might have problems since the next request might go to another server and therefore will not find that cookie data. That data needs to be stored in a database where all servers have access to it.

So, hopefully that is enough to let you know that in order to create a framework that scales, you really just need to understand scaling, and build or use your desired framework in a way that does not make any decisions that would afflict your sites ability to scale. Then, it’s up to the user of the framework to scale it.

You could go a different direction and make the framework force the user into using it’s own APIs for everything, and write the APIs in a way that you KNOW will scale in certain situations. Then, have the user set up the site in that certain situation. This is similar to how Google App Engine works. While a neat idea, this does restrict the user into using what Google allows.