Of course, Xcode provides “Groups” to organize your source code, but the idea is to be able to quickly identify the different kind of files that make up your Xcode project, either for Mac or for the iPhone, without having to open the Xcode project file. This means having both a folder structure, and an internal source code file structure. All of this will help you maintain your project in the future, which means cheaper costs, and less time spent looking for bugs.

All of this is also particularly useful when browsing projects via Google Code, Github or any other kind of file view of source code repositories. If your code is organized in a nice folder structure, it is easier to explore than if all the files sit in the same folder.

In this post I will enumerate some best practices that I use in all of my projects. So let’s say that you start a new Xcode project. Here’s the Xcode window that is presented to you (seen in “Condensed” mode, which is the one I prefer):

This is the project layout as seen in the Finder:

As you can see (and as you might have experienced), in the default layout used by Xcode, all new source code files will be thrown into the “Classes” folder, while all the new Resources will be just stored in the project root. In the long term, this layout can be really painful to deal with. So let’s just start rearranging things a bit:

Organize source files in folders and mirror them in Xcode

When I start a new Xcode project, I usually do the following:

1. I remove the “Classes” group (just deleting references, not moving the items to the trash, as shown in the image below)
2. Then I add the following subfolders to the “Classes” folder:
   • AppDelegate
   • Controllers
   • Models
   • Helpers
3. Finally, drag the enhanced “Classes” folder from the Finder to the Xcode project window, asking to “recursively create groups for every subfolder”:

Create separate folders for different Resource elements

The next step is to actually create a resource folder in Finder, and add a subfolder for every kind of resource that my project will use: sounds, images, SQLite databases, NIBs, etc. Then I do the following:

1. Remove the Resource group from the Xcode project (just deleting references)
2. Then I drag the newly created “Resources” folder from the Finder to the Xcode project window, asking to “recursively create groups for every subfolder”, like we did for the “Classes” folder.

Doing this has an interesting side effect: when you localize your application in other languages, each folder will contain a subfolder with the localized resources inside (for example, an “en.lproj” for English, “es.lproj” for Spanish, and so on).

Organize your code consistently

Each @implementation *.m file should always present methods in this order:

1. init and dealloc
2. public methods
3. public @dynamic properties
4. delegate methods (for each supported protocol)
5. private methods

Use #pragma statements to separate the regions shown above

Each logic group of methods should be separated from each other using the following lines (just type “#p” and hit the TAB key in Xcode!): [source:c:firstline(79)] // … return cell; }

pragma mark -

pragma mark UIAlertViewDelegate methods

• (void)alertView:(UIAlertView *)alertView clickedButtonAtIndex:(NSInteger)buttonIndex { // … [/source] The advantage of this approach is that later, you can use those #pragma marks to generate an automatic layout in the symbols pop-up of Xcode:

You can get this pop-up window clicking on this sector of your Xcode window:

Only leave public methods in the header files

This means putting private methods definitions in a (Private) category on top of the *.m file. This will remove all compiler warnings (about “this class might not respond to this selector”) and will cleanly separate what’s public from what’s not: [source:c:firstline(9)]

import “UntitledViewController.h”

@interface UntitledViewController (Private) - (id)returnPrivateObject; - (void)changeInternalState:(NSString *)param; @end

@implementation UntitledViewController

• (id)init { [/source]

Use consistent coding conventions

You can use my own, if you wish.

Treat warnings as errors

I’ve said a lot about that in a previous post, but here’s a quick reminder:

Create “Distribution” configurations for the new project

Duplicate the “Release” configuration and create two new ones: “Distribution Ad Hoc” and “Distribution App Store”. Each one will have to be configured with their corresponding provisioning profiles.

Add an “Entiitlements.plist” file to the project

Remember to uncheck (disable, turn off) the “get-task-allow” value (I still don’t understand why every Xcode project does not create this file automatically). Then add the “Entitlements.plist” value to the corresponding key in the “distribution” configurations created in the previous step.

Use source control

As soon as your project source tree is ready, commit it to your repository, whichever this is.

Conclusion

This is how the final project might look like:

Of course, you might as well enforce the above best practices using your own default project templates; depending on your requirements, this might be a useful thing to do. You must store those new Xcode templates in the following locations:

• iPhone: /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Project Templates
• Mac: /Developer/Library/Xcode/Project Templates

Hope this helps! As usual, feel free to add your comments, best practices, rants and other reactions in the comments section below.

One of the most famous citations in the book is the one regarding the consequences of adding human resources to a late project; this article will provide a couple of thoughts about this assertion, and highlight some contrariwise opinions. The Mythical Man-Month

The second chapter of Brooks’ masterpiece is named exactly as the book, “The Mythical Man-Month”; the core argument of this chapter is that the most frequent factor of project failure is schedule and time estimation. Brooks states that this is due to the fact that

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them. This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming. When a task cannot be partitioned because of sequential constraints, the application of more effort has no effect on the schedule. The bearing of a child takes nine months, no matter how many women are assigned.

(Brooks, pages 16 & 17)

The final phrase of the above paragraph is often used as a graphical depiction of the nature and meaning of Brooks’ law. It implies the strong need for communication and integration existing in software projects; being social processes, software requires a strong network of communication between team members, allowing them to coordinate the inherent set of interdependencies that every project has.

After an interesting analysis of common time overrun situations, Brooks ends this chapter with the following conclusion, which contains the enunciation of the law itself:

Oversimplifying otrageously, we state Brooks’s Law: Adding manpower to a late software project makes it later. This is then the demythologizing of the man-month. The number of months of a project depend upon its sequential constraints. The maximum number of men depends upon the number of independent subtasks. From these two quantities one can derive schedules using fewer men and more months. (The only risk is product obsolescence.) One cannot, however, get workable schedules using more men and fewer months. More software projects have gone awry for lack of calendar time than for all other causes combined.

(Brooks, pages 25 & 26)

This “law” is known and cited throughout the industry as an example of a common pattern, observed once and again in different projects all over the world:

Fact 3: Adding people to a later project makes it later(…) Intuition tells us that, if a project is behind schedule, staffing should be increased to play schedule catch-up. Intuition, this fact tells us, is wrong. The problem is, as people are added to a project, time must be spent on bringing them up to speed.(…) Furthermore, the more people there are on a project, the more the complexity of its communication rises.

(Glass, page 16)

As a personal experience, I must say that the lecture of this book opened my eyes more than many, many other books. It is a funny read, but also an enlightening one: many anecdotes told by Brooks strangely correspond to my own experience, and this one is no exception. I have seen projects gone unfortunately late because of the simple fact of adding more people; and in one particular case, the project was cancelled altogether. These projects had several factors in common, though:

• Bad documentation, or lack thereof; the only way for newcomers to the project to know what was going on was interrupting the other developers, disrupting the current operations on the project; I think that a good set of documents, describing both the high-level architecture and the low-level APIs are needed for new developers to jump in and catch up. It’s maybe not enough, but a good leap forward anyway.
• Lack of architectural vision; projects that do not have an architect, providing vision and technical leadership to the team, are in my opinion exposed to problems when more developers join the project. The architect can act as a proxy person, guiding new developers while they familiarize themselves with the project, isolating other developers from this task.
• Bad project decomposition in components; if the system to be developed is sufficiently large, and the decomposition in components is not properly done, the overlap and extended communication paths among team members might affect the whole project negatively. A good decomposition breaks down the whole project in a set of smaller ones, with the corresponding set of interfaces, which brings the whole team to work separately on different subsystems. In these, the risk of getting later for adding manpower is reduced proportionally.
• Bad working conditions; I positively think that open spaces are a common disease in our industry. Teams working in open spaces suffer more of noise and visual distractions, and this is more evident when new team members join the project.

Criticism

However famous, Brooks’ law has had a good deal of criticism as well, regarding the specific characteristics of projects that might be affected in case that new people is assigned to them. The OS/360 project, which served as the basis for Brooks’ work, might not be similar to other projects, and as such, the law would not necessarily apply to them:

For Brooks’ Law to be true, the amount of training effort required from existing staff must be significant. The amount of effort lost to training must exceed the productivity contributed by new staff when they eventually become productive. (…) “Late” chaotic projects are likely to be much later than the project manager thinks–project completion isn’t three weeks away, it’s six months away. Go ahead and add staff. You’ll have time for them to become productive. Your project will still be later than your plan, but that’s not a result of Brooks’ Law. It’s a result of underestimating the project in the first place.(…) Controlled projects are less susceptible to Brooks’ Law than chaotic projects. Their better tracking allows them to know when they can safely add staff and when they can’t. Their better documentation and better designs make tasks more partitionable and training less labor intensive. They can add staff later in the project with less risk to the project.

(McConnell, 1999)

Scott Berkun gives a more concrete analysis on why the law could be wrong:

• It depends who the manpower is. The law assumes that all added manpower is equal, which is not true.
• Some teams can absorb more change than others. Some teams are more resiliant to change.
• There are worse things than being later. (…) That can be ok if you also get higher quality
• There are different ways to add manpower. (…) The more experience everyone has with mid-stream personnel changes, the better.
• It depends on why the project was late to begin with. (…) no amount of programming staff modifications will resolve the psychiatric needs of team leaders or the dysfunctions of executives.
• Adding people can be combined with other management action. (…) if you’re removing your worst, and most disruptive, programmer and adding one of your best, it can be a reasonable choice.

(Berkun, 2006)

And what about open source projects? Many of these (Linux, Apache, MySQL) are potentially among the biggest software projects ever undertaken, and they don’t appear to suffer o fthe problems pictured by Brooks’ law:

But proponents of open source and free software development, including Linux developers, are not completely satisfied with the Law. Most famously (among geeks at any rate), Eric Raymond in his “The Cathedral and the Bazaar,” declared Brooks’ Law obsolete, if not simply limited, saying “if Brooks’ Law were the whole picture, Linux would be impossible.” Although Raymond now says that he has somewhat modified his views or was misunderstood, some still would say he is given to oversimplifying and outrageousness himself. “I don’t consider Brooks’ Law ‘obsolete’ any more than Newtonian physics is obsolete; it’s just incomplete. Just as you get non-Newtonian effects at high energies and velocities, you get non-Brooksian effects when transaction costs go low enough. Under sufficiently extreme conditions, these secondary effects dominate the system — you get nuclear explosions, or Linux.”

(Jones, 2000)

Conclusion

So far, the discussion seems to be open. There might be a scale factor for projects, which in turn might expose them to be affected by Brooks’ law. I think that research is needed to arrive to a conclusion, even if it will be a statistical one.

Other important facts highlighted in the book are the “second system phenomenon”, the productivity advantage of using high-level languages, and the importance of building a prototype – “one to throw away”. I can only recommend this book to everyone interested in the field of software engineering (which I did in my own review of classic books in this blog: http://kosmaczewski.net/2005/11/20/my-bookshelf-part-iii/ )

References

Berkun, S.; “Exceptions to Brooks’ Law”, January 11th, 2006, [Internet] http://www.scottberkun.com/blog/2006/exceptions-to-brooks-law/ (Accessed June 8th, 2007)

Brooks Jr., F. P.; “The Mythical Man-Month – Essays on Software Engineering, Anniversary Edition”, 1995, Addison Wesley, ISBN 0-201-83595-9

Glass, R. L.; “Facts and Fallacies of Software Engineering”, Addison-Wesley, 2003, ISBN 0321117425

Jones, P.; “Brooks’ Law and open source: The more the merrier?”, IBM, May 1st, 2000, [Internet] http://www.ibm.com/developerworks/linux/library/os-merrier.html (Accessed June 8th, 2007)

McConnell, S.; “Brooks’ Law Repealed?”, IEEE Software, November/December 1999 [Internet] http://stevemcconnell.com/ieeesoftware/eic08.htm (Accessed June 8th, 2007)

In this article I will provide a short overview of two kinds of challenges that I consider that software engineers will have to confront in the next 20 years: the human and the technical. The Human Factor

A quick look at the agenda of the 29th Int. Conference on Software Engineering (held in Minneapolis last year, from the 20th to the 26th May 2007) shows the key themes considered by the software engineering research community as the major challenges today:

• “Improving Software Practice through Education: Challenges and Future Trends”
• “Research Collaborations between Industry and Academia”
• “Model-driven Development of Complex Systems: A Research Roadmap”
• “Source Code Analysis: A Road Map”
• “Software Reliability Engineering: A Roadmap”
• “Global Software Engineering: The Future of Socio-technical Coordination”
• “Collaboration in Software Engineering: A Roadmap”
• “Self-Managed Systems: An Architectural Challenge”
• “Software Project Economics: A Road Map”

(Source: ICSE 2007)

Mixed up with technical concerns, some presentations highlighted core problems that appears in the current state of software engineering: communication, collaboration and human issues.

The core substance of software deserves more eyes and more minds, thinking ways to describe not only the big picture (something that you can do with fancy diagrams) but also to give solutions to the problems that developers find daily while building systems up. Software is a process, but not any kind of process: a human one, maybe the most intangible of all processes; and as such, it is filled with all human brightnesses and failures.

(Myself, in 2006)

I have the deep, strong conviction that software development cannot and must not be separated from the human-side problems of forming, keeping and training teams, enhancing the internal and external communications, improving and enhancing the individual creativity as well as the ways of reaching team consensus. As a powerful example, the seminal Peopleware book by DeMarco and Lister showed that many of the most successful software companies have been those that excelled in creating human-centric environments:

In 1982, (Mitchell Kapor) founded Lotus Development Corporation, for which he is most noted. While there, he revolutionized corporate workplace culture by making diversity and inclusivity top priorities in his goal for creating an environment that attracted and retained employees. There were many “firsts” for Lotus, including being the first company to sponsor an AIDS Walk event in the mid-80′s and refusing to do business with South Africa due to Apartheid.

(Sterling-Hoffman)

Thanks to a sharp hiring process, a series of innovations in their flagship spreadsheet product, and a progressive corporate culture, Lotus dominated the software landscape of the 80s. Today, Google follows very closely Lotus’ steps (Google, 2007a), and their brilliant results in the last few years seem to confirm this trend. Google for example allows their employees to use 20% of their time in their own projects (Google, 2007b). This is resulting in an incredible amount of code, used internally and also released as open-source projects:

Google is a fantastic company to work for. I could cite numerous reasons why. Take the concept of “20 percent time.” Google engineers are encouraged to spend 20 percent of their time pursuing projects they’re passionate about. I started one such exciting project some time back, and I’m pleased to announce that Google is releasing the fruits of this project as an open source contribution to the Macintosh community. That project is MacFUSE, a Mac OS X version of the popular FUSE (File System in User Space) mechanism, which was created for Linux and subsequently ported to FreeBSD.

(Google Mac Blog, 2007)

The empowerment of both the individual and the team (the emphasis is important here) is key for a successful software project.

Parallelization

Herb Sutter has put it very clearly: technically speaking, since the beginning of the decade, there is no way for getting more processing power without jumping to multicore architectures:

The key question is: When will it end? After all, Moore’s Law predicts exponential growth, and clearly exponential growth can’t continue forever before we reach hard physical limits; light isn’t getting any faster.(…) If you’re a software developer, chances are that you have already been riding the “free lunch” wave of desktop computer performance.(…) Right enough, in the past. But dead wrong for the foreseeable future.

(Sutter, 2005)

The problem is that more cores do not necessarily mean more computing power, because the jump done by chip manufacturers has not (yet) been completely followed by the software community. Of course there is the concept of “threads”, and multi-threaded applications can benefit of performance boosts when running on multicore hardware platforms; however, a number of myths have to be debunked, as the common “2 x 3GHz = 6GHz” (as explained by Sutter here: http://www.ddj.com/showArticle.jhtml?documentID=ddj0503a&pgno=3), and even more importantly, creating multithreaded applications is not easy. At all.

A couple of months ago, in the “Questions and Answers” of LinkedIn.com I answered an interesting question about parallelization; the following excerpt of my answer pretty much summarizes my opinions about the current state of multithreading, as well as some challenges that are raised for the future:

The problem is simply that the “streamline” programming languages languages do not provide good ways to code multithreaded applications. (…) Not at all. The problem is real, since multithreading applications are extremely complicated to think of, let alone develop properly. A line of code in a high-level language could mean several hundred instructions in a processor; and depending on the sharing algorithm used at the CPU level, each one of these instructions might be executed separately, sharing resources with other processes. So what happens when? (…) What I mean is that the fact that the JVM and the CLR support threads does not make good .NET or Java developers good multithreading developers by default. It’s a different mindset; who is accessing your resources? (…) I think that as long as programming languages do not take multitasking and multithreading as base features (and not as mere library or API add-ons) we will continue struggling with single-threaded applications that collide with each other.

(Myself, this time on LinkedIn Answers, 2007)

I think that the challenge of parallelization is not only an extremely tough one, requiring what Thomas Kuhn calls a “paradigm shift”, but also an extremely huge business opportunity; after all, while the top of the Chinese ideogram for “Crisis” means “Danger”, the bottom part means “Opportunity” (Mary R. Bast, 1999).

Very Large Systems

I also think that software systems will invariably get bigger and bigger. And given the historically high risk of failure of software projects, the dependency on software of the modern society, the pervasiveness of the Internet, the low prices of connectivity and the overall globalization, it is more important than ever to get ready for those challenges.

In July 2006, the well known Software Engineering Institute of the Carnegie Mellon University published an impressive report (freely downloadable) called “Ultra-Large-Scale (ULS) Systems: The Software Challenge of the Future”:

The study brought together experts in software and other fields to answer a question posed by the U.S. Army Office of the Assistant Secretary of the U.S. Army (Acquisition, Logistics & Technology): “Given the issues with today’s software engineering, how can we build the systems of the future that are likely to have billions of lines of code?” Increased code size brings with it increased scale in many dimensions, posing challenges that strain current software foundations. The report details a broad, multi-disciplinary research agenda for developing the ultra-large-scale systems of the future.

(SEI, CMU, 2006)

The 150-page long report gives an extremely detailed vision of the challenges raised by complex systems, in the following areas:

• Design
• Monitoring
• Human interaction
• Computational Engineering
• Deployment
• Legal issues

The report provides interesting conclusions, highlighting the methodologies and techniques that will required to tackle these systems efficiently, among them the role of the W3C, the forthcoming trends of grid computing and parallelization, the Model-Driven Architecture (MDA) initiative of the OMG, and finally the development of larger Service-Oriented Architectures (SOA) platforms, such as .NET or J2EE (page 41 of the report).

The report also places a strong emphasis in the concept of socio-technical ecosystems and I think it’s worth a read by everyone interested in software engineering.

Conclusion

Given its youth, we have yet to see the most important developments in software engineering. However, it is extremely difficult to predict the future in this industry: Bill Gates himself published a book in 1995, “The Road Ahead”, where he only slightly talks about the World Wide Web:

“The Road Ahead” appeared in December 1995, just as Gates was unveiling Microsoft’s master plan to “embrace and extend” the Internet. Yet the book’s first edition, with its clunky accompanying CD-ROM, mentioned the Web a mere seven times in nearly 300 pages. Though later editions tried to correct this gaffe, “The Road Ahead” remains a landmark of bad techno-punditry — and a time-capsule illustration of just how easily captains of industry can miss a tidal wave that’s about to engulf them.

(Salon.com, 2000)

In any case, I think that there are important challenges in our industry: the need for better human management, the jump to multicore architectures and multiprocessing, and the ever-growing size of software projects. These three elements will without any doubt change the shape of the industry in the years to come, and raise new challenges in turn.

References

Adrian Kosmaczewski on LinkedIn Answers, “For the software architects out there, do you feel there is an impending paradigm shift in the software development model, towards “parallel computing” models?”, January 2007, [Internet] http://www.linkedin.com/answers?viewQuestion=&questionID=7804&askerID=4194838 (Accessed June 3rd, 2007)

Adrian Kosmaczewski on Kosmaczewski.net, “What will the Software Architecture discipline look like in 10 years’ time?”, March 16th, 2006 [Internet] http://kosmaczewski.net/2006/03/16/software-architecture-future/ (Accessed June 3rd, 2007)

Bast, Mary R; “Crisis: Danger & Opportunity”, 1999 [Internet], http://www.breakoutofthebox.com/crisis.htm (Accessed June 3rd, 2007)

DeMarco, Tom & Lister, Timothy, “Peopleware – Productive Projects and Teams, 2nd Edition”, 1999, Dorset House Publishing, ISBN 0-932633-43-9

Google, “Top 10 Reasons to Work at Google”, 2007a [Internet] http://www.google.com/jobs/reasons.html (Accessed June 3rd, 2007)

Google, “What’s it like to work in Engineering, Operations, & IT?”, 2007b, [Internet] http://www.google.com/support/jobs/bin/static.py?page=about.html (Accessed June 3rd, 2007)

Google Mac Blog, “Taming Mac OS X File Systems”, January 11th, 2007, [Internet] http://googlemac.blogspot.com/2007/01/taming-mac-os-x-file-systems.html (Accessed June 3rd, 2007)

ICSE, “Future of Software Engineering”, 2007, [Internet] http://web4.cs.ucl.ac.uk/icse07/index.php?id=104 (Accessed June 3rd, 2007)

Software Engineering Institute, Carnegie-Mellon University, “Ultra-Large-Scale (ULS) Systems – The Report”, July 2006, [Internet] http://www.sei.cmu.edu/uls/ (Accessed June 3rd, 2007)

Salon.com, 2000 [Internet], “Why Bill Gates still doesn’t get the Net”, [Internet] http://archive.salon.com/21st/books/1999/03/cov_30books.html (Accessed June 3rd, 2007)

Sutter, Herb; “A Fundamental Turn Toward Concurrency in Software”, 2005, [Internet] http://www.ddj.com/dept/architect/184405990 (Accessed June 3rd, 2007)

Sterling-Hoffman, “Opening Doors To Higher Education”, [Internet] http://www.sterlinghoffman.com/newsletter/articles/article140.html (Accessed June 3rd, 2007)

Wikipedia, “Thomas Kuhn” [Internet], http://en.wikipedia.org/wiki/Thomas_Kuhn (Accessed June 3rd, 2007)

As you might know (and if you don’t I’ll tell you anyway), Django’s views (somehow occupying the “Controller” level in an MVC architecture) must take (at least) an HttpRequest instance as a parameter and must return an HttpResponse instance. That’s how it goes in Django, this is the law. This means that you must be sure that the last instruction in your request processing code (in whichever way you’ve organized it) must return an HttpResponse instance, usually calling the HttpResponse() constructor (or of any of its useful subclasses), or by calling the django.shortcuts.render_to_response() function, or something similar.

This has, in my opinion, a major drawback: it might limit code reuse and it increases the coupling in the code. Everything’s not lost, however.

Before you start the flame wars, let me explain, using an example coming from the Django website; this represents a basic Django view function, returning some response containing data fetched from the database:

[source:python] from django.shortcuts import render_to_response, get_object_or_404

…

def detail(request, poll_id): p = get_object_or_404(Poll, pk=poll_id) return render_to_response(‘polls/detail.html’, {‘poll’: p}) [/source]

Let’s say now that I want to reuse that particular data (the ‘p’ variable) in another view: given that the return value is always an HttpResponse instance, you are screwed; sometimes you just need the data, to find something, or simply to render it in another format like JSON or XML (RESTful architectures, anyone?). This goes pretty much against the DRY principles, and if you don’t go deeper than the Django tutorials, your whole application might feature lots of repeated code.

Even worse, you have a direct reference to a template (“polls/detail.html”), and this kind of coupling does not scale well. It can become a real problem in big projects.

There are, however, strategies to avoid this: the first, the most common, is to refactor your code and to create a “layer” of data-specific functions, which will return instances (or arrays thereof) that you can reuse here and there. Doing this in a big project already started requires a good deal of unit testing first, to ensure that your refactoring is not breaking something elsewhere, but that’s another problem (because you DO unit test, right??). This approach might not scale well in complex projects, and thus you would like to organize your code in other ways.

I learnt about organizing views using callable objects instead of functions while studying the code in the Django REST Interface project. In this case, you create code like this:

[source:python] class Resource(ResourceBase): “”" Generic resource class that can be used for resources that are not based on Django models. “”"

# ... snip ...
def __call__(self, request, *args, **kwargs):
    """
    Redirects to one of the CRUD methods depending
    on the HTTP method of the request. Checks whether
    the requested method is allowed for this resource.
    """
    # Check permission
    if not self.authentication.is_authenticated(request):
        response = HttpResponse(_('Authorization Required'), mimetype=self.mimetype)
        challenge_headers = self.authentication.challenge_headers()
        response._headers.update(challenge_headers)
        response.status_code = 401
        return response
    try:
        return self.dispatch(request, self, *args, **kwargs)
    except HttpMethodNotAllowed:
        response = HttpResponseNotAllowed(self.permitted_methods)
        response.mimetype = self.mimetype
        return response

[/source]

The important bit here is the “call” method, which allows an instance to be called as a function, without specifying any particular method. This makes me remember of the dreadful VB default methods but in Python it’s not that bad, actually (VB is horrible by default anyway), and allows you to use a cool syntax to do complex tricks (“command pattern” way of doing things, without the method call overload). And of course, since you are using an object-oriented approach, you can use polymorphism and inheritance to organize and reuse code as much as you can (or want).

Finally, Marco told me that his team uses another cool approach: they avoid returning HttpResponse instances from the views, and instead use Python decorators to generate those. This way, you can achieve another neat separation of concerns, and you can reuse code simply and effectively.

I understand that the Python philosophy cares about explicitness, but the “easy” way of processing requests in Django leads to trouble in big applications: increased coupling, reduced DRY, more headaches. I think you should use some code-reuse strategy in your Django code, but this, of course, is more an architectural problem than a Django problem.

Charles Petzold, in its book “Code”, states the following:

In theory, application programs are supposed to access the hardware of the computer only through the interfaces provided by the operating system. But many application programmers who dealt with small computer operating systems of the 1970s and early 1980s often bypassed the operating system, particularly in dealing with the video display. Programs that directly wrote bytes into video display memory ran faster than programs that didn’t. Indeed, for some applications – such as those that needed to display graphics on the video display – the operating system was totally inadequate. What many programmers liked most about MS-DOS was that it ‘stayed out of the way’ and let programmers write programs as fast as the hardware allowed.

(Charles Petzold, “Code”, pages 332 & 333)

This paragraph shows the state of things during the MS-DOS & early Windows versions timeframe (from late 1970s until 2000 approximately). During this time, programmers could directly access computer memory, bypassing the APIs offered by the operating system, and thus having total control of the hardware.

This shows two different trends in computer programming, one that respects the functionality offered by the operating system, and another that bypasses it. There are advantages and disadvantages to each approach, and the following paragraphs shows some of them. The Conflict

The Apple Macintosh (1984) and the NeXT computer (1989) were among the first systems to introduce a complete API (Application Programming Interface) that completely shielded application developers from directly accessing the hardware on which the application ultimately runs. In the case of the Apple Macintosh, this API could be programmed in Pascal, while for the NeXT it was using the Objective-C language.

The tradeoff between the MS-DOS approach and the API one can be resumed in three areas: performance, portability & maintenance, and security. The first one, considered critical in the 80s, has been one of the major factors of the success of the “compatible IBM PC” + MS-DOS platform; similar applications could ran faster than in Macintosh environments; also, a bigger number of games (which heavily use graphics) was available for that platform, and this also led to a majority of people to choose it.

Direct access to the hardware brought a first problem, of maintenance and portability; indeed, software written this way was too hard to port to different operating systems or processor architectures, since it relied heavily in the availability of certain hardware interruptions and circuitry. While, on the other hand, Apple Macintosh software created for the first version of the Mac OS (1984) could run seamlessly, without recompilation (just copy the executable and double-click on it), until Mac OS 9 (1999).

Microsoft Windows

Regarding Microsoft Windows, the situation is slightly more complicated. Windows began its life as a GUI around MS-DOS in 1985, and from its version 95 it gradually became a more independent system, but never truly becoming a multi-threaded, multi-tasking operating system. This system used to support old MS-DOS software, allowing it to run natively:

I first heard about this from one of the developers of the hit game SimCity, who told me that there was a critical bug in his application: it used memory right after freeing it, a major no-no that happened to work OK on DOS but would not work under Windows where memory that is freed is likely to be snatched up by another running application right away. (…) They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it. This was not an unusual case. The Windows testing team is huge and one of their most important responsibilities is guaranteeing that everyone can safely upgrade their operating system, no matter what applications they have installed, and those applications will continue to run, even if those applications do bad things or use undocumented functions or rely on buggy behavior that happens to be buggy in Windows n but is no longer buggy in Windows n+1. In fact if you poke around in the AppCompatibility section of your registry you’ll see a whole list of applications that Windows treats specially, emulating various old bugs and quirky behaviors so they’ll continue to work.

(Joel Spolsky, 2004)

As you can see, direct hardware access is not only a problem for applications developers… it was one for Microsoft as well.

Simultaneously, Windows NT was started in 1988 by another team, largely composed of engineers that worked in the Digital Equipment Corporation OpenVMS system. The NT kernel features, among several distinctive characteristics, the HAL (Hardware Abstraction Layer):

A hardware abstraction layer (HAL) is an abstraction layer between the physical hardware of a computer and the software that runs on that computer. Its function is to hide differences in hardware and therefore provide a consistent platform to run applications on.

(Wikipedia, 2006)

The NT’s HAL abstracts the complete hardware beneath, and the only way to access hardware functionality is through the API. The NT team approach was radically different to that of the “classic” Windows team; they would not fix compatibility issues application per application, but would rather define an API upfront and publish it to the developers.

The NT kernel has since replaced the older 9x kernel, from version XP onwards. This shift broke many old software packages, that are not able to run properly (if at all) in the new versions of Windows. Only those programs that use the Windows API exclusively are able to run properly (I have a copy of Lotus Improv 3.1, bought in 1992, that I used to run under Windows 3.1, and that runs perfectly well under Windows XP…!)

Present Situation

Nowadays the “performance” characteristic named above has been replaced by the security concern; code that can access directly the computer hardware without permission checks (particularly in times of the Internet) can be potentially extremely dangerous: chapter 4 of the book “Hacking Exposed” (ISBN 0-0721-2127-0), by Joel Scambray, Stuart McClure and George Kurtz exposes tens of different vulnerabilities caused by direct hardware access known to Windows 95, 98 and ME – that is, the non-NT kernel. In other terms, such systems connected directly to the Internet are simply too vulnerable to be safely usable.

The conflict between calling an API or accessing the hardware directly today has been won by the API approach; consumer operating systems, at least, have taken this road and there is no turning back. The advantages are evident; the same source code base can be used to build the same application in several different platforms, lowering maintenance and support costs; platform vendors can improve performance and security reducing the impact in existing software packages; application developers can share knowledge, tips and tricks; security is built from the ground up.

On the other side, the growth in capacity of operating systems make limited resources to appear as unlimited: memory (using paging on disk), printers (using print queues) or even screen desktop space (using multiple desktops such as the GNOME approach, or on-screen gadgets such as Expose on the Mac).

Conclusion

Currently there are APIs for graphic manipulation, such as OpenGL, that allow software such as Google Earth to run in different hardware architectures using the same code base. This is possible thanks to the higher power of today’s hardware, and to the advances in operating system design, that make the overhead of API calls a non-significant portion of the overall CPU time needed to execute programs. Thus, the conflict has largely been resolved, in my opinion.

References

Charles Petzold, “Code – The Hidden Language of Computer Hardware and Software”, 2000, Microsoft Press, ISBN 0-7356-1131-9 (website)

Joel Spolsky, “How Microsoft Lost the API War”, June 13, 2004 [Internet], http://www.joelonsoftware.com/articles/APIWar.html, (Accessed December 14th, 2007)

Wikipedia, “Hardware abstraction layer” [Internet], http://en.wikipedia.org/wiki/Hardware_abstraction_layer (Accessed December 14th, 2007)

Modern operating systems provide a clear separation of the kernel processes from those running in user space, which prompts the question of how to access I/O devices from user processes, without breaking the above mentioned architectural separation, which guarantees stability, security and performance.

Several approaches are available, depending on the level of abstraction used and the context of the problem. One of the solution is using POSIX Device Files, which indeed provide the same system-call interface for both files and devices. This article will describe the POSIX standard, the POSIX device files, and give a short enumeration of advantages and disadvantages of them. The POSIX Standard

POSIX stands for “Portable Operating System Interface for Unix”, and it is the “collective name of a family of related standards specified by the IEEE to define the application programming interface (API) for software compatible with variants of the Unix operating system.” (Wikipedia). Nearly all of today’s most important operating systems, in either closed or open source form, are POSIX-compliant to a certain degree: all versions of Microsoft Windows, Apple Mac OS X, OpenVMS and Solaris, for example, are fully POSIX-compliant, while Linux, FreeBSD and Nucleos RTOS are partially compliant (Wikipedia). POSIX Conformance Test Suites (for example the one provided by NIST) are used to determine the level of POSIX-compliance of an operating system.

The importance of the POSIX standard is hard to assess; however, it is widely recognized that it has had an enormous impact in the computing world as we know it today:

The overall Unix market (Figure 1) boasts billions of dollars in revenues, and is projected to continue growing into the future. Major areas for this growth feature high-end systems (data warehousing, servers, and supercomputing) rather than desktop applications. Brian Unter evaluated the impact of Posix standards in this area as responsible for up to 30% of Hewlett-Packard’s Unix business.

(Isaak & Johnson, 1998)

The POSIX standard provides the whole industry with a single, enormous and strong foundation, allowing interoperability, benchmarking and standardization, which in turn helps reducing costs, making the whole industry more efficient.

POSIX Device Files

One of the key aspects of the POSIX standard is the definition of a common gateway to access I/O devices from the user space, avoiding a direct and probably dangerous communication with the kernel, and known as POSIX Device Files: “These interfaces enable communication with serial, storage, and network devices through device files. In any UNIX-based system such as BSD, a device file is a special file located in /dev that represents a block or character device such as a terminal, disk drive, or printer. If you know the name of a device file (for example, disk0s2 or mt0) your application can use POSIX functions such as open, read, write, and close to access and control the associated device.” (Apple)

What does all of this mean? In short, this means that

Under UNIX, every piece of hardware is a file. To demonstrate this, try view the file /dev/hda less -f /dev/hda /dev/hda is not really a file at all. When you read from it, you are actually reading directly from the first physical hard disk of your machine. /dev/hda is known as a device file, and all of them are stored under the /dev directory.

(Sheer)

Advantages

The main advantage is homogeneity. Using POSIX device files, developers can manipulate, read and write data from external I/O devices without having to directly access the operating system kernel, using a unified programming model, with a standardized set of semantics, which leads to systems designed with elegance and correctness.

Another advantage is tightly related to the concept of “streams”, as understood by many higher-level programming languages. Streams allow developers to access devices in an uniform way; C++, Java, the .NET languages, and even dynamic languages such as Ruby or Lisp allow to treat sequences of bytes using this uniform concept:

A stream is an object that can be used with an input or output function to identify an appropriate source or sink of characters or bytes for that operation

(Lisp.org)

Java and .NET have big inheritance trees below the abstract Stream class, handling memory-based, file, network or I/O device-based streams, all of them sharing a similar structure and behavior, this means mainly the capability of being read and / or written in a sequential fashion.

This paradigm, coupled with the use of POSIX device files, allow software architects to create flexible designs, where the streams are treated polymorphically, can be exchanged at runtime as needed, and have lesser dependencies among them.

Disadvantages

The main disadvantage might be the maintainability of the code that uses these abstractions, since that these notation and semantics might not be immediately obvious to a newly graduated programmer.

The most common programming experience, in higher-level abstractions, usually depends on higher abstractions to access devices, for example using the System.IO.SerialPort classes in the .NET programming model:

To reduce the programming effort that is required when working with serial ports, the .NET Compact Framework 2.0 includes the SerialPort class. The SerialPort class provides a simplified abstraction over serial communications ports that provides a number of features that simplify monitoring and configuring serial ports. The serial port also simplifies sending and receiving data with serial ports—including the automatic encoding and decoding of data sent to and received from the port

(Microsoft)

Mac OS X also provides higher-level abstractions for managing devices:

A device interface is a plug-in interface between the kernel and a process in user space. The interface conforms to the plug-in architecture defined by Core Foundation Plug-in Services (CFPlugIn), which, in turn, is compatible with the basics of Microsoft’s Component Object Model (COM). In the CFPlugIn model, the kernel acts as the plug-in host with its own set of well-defined I/O Kit interfaces, and the I/O Kit framework provides a set of plug-ins (device interfaces) for applications to use

(Apple)

The abstraction-level problem also arises when porting code from an operating system to another, since a common abstraction to access a CD-ROM drive in Windows (usually “D:\”) is different from the way that Linux does (usually “/cdrom”) or the way that Mac OS X uses (which is using the friendly name of the CD-ROM under the “/Volumes” device). The use of POSIX device files might help avoiding these problems.

Conclusion

Both the advantages and disadvantages of using the same system-call interface are relative to the problem being solved, the available budget and the skills of the engineering team. There are huge advantages to stick to the POSIX standard, but there is a cost in maintainability and readability as well.

References

Apple Developer Connection, “I/O Kit Fundamentals, Architectural Overview: Controlling Devices From Outside the Kernel”, [Internet] http://developer.apple.com/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/ArchitectOverview/chapter_3_section_7.html (Accessed February 11th, 2007)

Isaak, J.; Johnson, L.; “Posix/Unix standards: foundation for 21st century growth”, Micro, IEEE, Volume 18, Issue 4, July-Aug. 1998 Page(s):88, 87; Digital Object Identifier 10.1109/40.710874

Lisp.org, “System Class STREAM”, [Internet] http://www.lisp.org/HyperSpec/Body/syscla_stream.html (Accessed February 11th, 2007)

Microsoft, “What’s New in the .NET Compact Framework 2.0″, [Internet] http://msdn2.microsoft.com/en-us/library/aa446574.aspx (Accessed February 11th, 2007)

NIST, “PCTS:151-2, POSIX Test Suite”, [Internet] http://www.itl.nist.gov/div897/ctg/posix_form.htm (Accessed February 11th, 2007)

Sheer, P.; “UNIX devices”, [Internet] http://www.cacs.louisiana.edu/~mgr/404/burks/linux/rute/node18.htm (Accessed February 11th, 2007)

Wikipedia, “POSIX”, [Internet] http://en.wikipedia.org/wiki/POSIX (Accessed February 11th, 2007)

Five years ago I worked as a Software Engineer for a startup, based in Geneva, Switzerland, which had the goal of creating a web-based systems management console, to control and monitor the status of large computer installations, much like Microsoft SMS (Systems Management Server) does. This tool would eventually benefit from being a web-based application, and as such it could be used from anywhere, without having to install a “fat client”; just launch a browser, point to a particular URL, and you are done.

During the project, I was able to work towards the creation of the first AJAX application I’ve ever seen (this was 2002!), and also, to use Aspect-Oriented Programming techniques for the first time.

The Architecture

This tool was designed as a three-systems project:

1. On the server side, the back-end: it used “software spiders” that crawled your local network for information about the domain computers, using their WMI (Windows Management Instrumentation) services to gather useful information such as processor type, network card address, memory size, CPU usage, any kind of information. This information was sent to the server, where it was stored in a SQL Server 2000 database, for quicker retrieval and reporting.
2. On the client side, a web based application, running on top of IIS (Internet Information Services) and implemented as an ASP application (Active Server Pages). This tool was the first AJAX (Asynchronous JavaScript and XML) application I had seen at the time (2002!), giving users a real-time experience of what was going on in the network; for example (this was impressive) you could just plug a new computer on the domain, and a couple of seconds later, the information would appear in your web browser automatically, without having to refresh the page (the client AJAX component polled the back-end every 5 or 10 seconds for updates).
3. In between both ends, the ASP application connected to the SQL Server backend using a COM+ “proxy” component, which was the only means of communication between both subsystems.

The development team was split in three teams, one working on the spider+database backend, and the other (where I was) working in the ASP client application. Finally, a highly-skilled C++ programmer had the task of building the “proxy” component that allowed both systems to communicate. This component was critical, and was designed for extreme high performance. Its design would deserve an article just for itself… just to mention, it featured transactional requests, complex event and feedback models and had extensive queuing capabilities, via MSMQ (Microsoft Messaging Queue).

Just for the record, regarding the methodologies used in the project, it is worth noting that the whole of the project was managed using the Rational Unified Process (RUP). It must be said that in the case of this startup company, this methodology was an “overkill”, and documentation was never kept in sync when changes were done to the architecture. Being a startup company required to be able to adapt the software as quickly as possible, and often this meant skipping the documentation process.

The DataServices Project

The COM+ component had extremely high throughoutput and small response times, but for the sake of interoperability, it required the ASP client application to send and consume complex XML requests and responses, using an industry standard called “CIM” (Computer Information Model). This XML standard, designed by the DTMF (Distributed Task Management Force) is part of the WBEM initiative allowing for XML-based interoperability of computer management systems. It is worth noting that the Microsoft Systems Management Server (SMS) is based on this standard as well, and that the Windows WMI services are also based on these standards.

The problem, for the web applications team, was then to create long and complex XML requests to send to the COM+ proxy component. The web developers, already struggling with a complex AJAX-based UI, had real trouble to figure out the different parameters for these calls, and to create the proper XML structure (The CIM XML format is an extremely complex tree structure, with several layers of information and a high level of redundancy). This led to longer development cycles, since every new use case implementation required a good deal of new requests to be created to the server, with low reuse of already existing requests.

This is how the “DataServices” subproject was born: to create a layer of abstraction between the COM+ component and the dynamic HTML application. The goal was to provide a simpler way to connect the backend with the user interface, so that the development team in the UI layer could concentrate in the user experience, rather than in creating long XML documents. This was dragging development times, and was a well-known bottleneck for the project.

To avoid impacting the current ASP application, and to leverage XML capabilities, this project was created as an ASP.NET application, using the recently released .NET Framework, version 1.0 (released in February 2002) and the C# language.

This ASP.NET application would handle requests from the dynamic HTML application, but providing an API that would be much simpler for developers; for instance, instead of having to create long XML strings to retrieve the list of processes running in a given computer of the domain, the developers could just call “ComputerSystem.GetRunningProcesses(machineName)” (“facade” methods) and that would do it.

Aspect-Oriented Programming

Once the DataServices subproject began growing, we noticed that we were performing the same operations everywhere, for each “facade” method call:

• Deserialization of requests
• Security checks
• Instrumentation (mainly logging, but also performance management and throughoutput)
• Back-end connection management (opening, pooling, closing)
• Error management
• Serialization of responses

The same lines of code were repeated around almost every internal operation done by the DataServices infrastructure, but this code could not be properly encapsulated in an external component, to be reused instead of duplicated.

More or less at the same time, one of the team members read a paper about the newly-born idea of “Aspect Oriented Programming” (AOP), and about the basic AOP capabilities of the .NET Framework (Dharma, Fell and Sells, 2002).

Gregor Kiczales from Xerox PARC, one of the creators of AOP, explains that

“We have found many programming problems for which neither procedural nor object-oriented programming techniques are sufficient to clearly capture some of the important design decisions the program must implement. This forces the implementation of those design decisions to be scattered throughout the code, resulting in “tangled” code that is excessively difficult to develop and maintain. We present an analysis of why certain design decisions have been so difficult to clearly capture in actual code. We call the properties these decisions address aspects, and show that the reason they have been hard to capture is that they cross-cut the system’s basic functionality. We present the basis for a new programming technique, called aspect-oriented programming, that makes it possible to clearly express programs involving such aspects, including appropriate isolation, composition and reuse of the aspect code.”

(Kiczales et al., 1997)

The following two images highlight the location, in the Apache Tomcat project, of different sections of code that handle URL pattern matching and logging:

(Source: Hilsdale, Kiczales, 2003)

(Source: Hilsdale, Kiczales, 2003)

The idea of AOP is to provide a mechanism to centralize these lines of code in a single location, which would provide better management and maintainability in the future; Jacobson further explains that

“Even though peer use cases are separate during requirements, their separation is not preserved during implementation. The realization of a use case touches many classes (scattering), and a class contains pieces of several use-case realizations (tangling). As a serious consequence, the realization of each use case gets dissolved in a sea of classes.”

(Jacobson, Ng, 2005)

(Source: Jacobson, Ng, 2005, page 33)

Regarding DataServices, suffice to say that AOP seemed to us as a good solution for our problem. Tempted by the discovery of a new paradigm, we decided to give a try to AOP, and the results proved to be worth the effort. In our case, we intercepted each message sent to the classes implementing the use cases (like the ComputerSystem.GetRunningProcesses(machineName) example above), “injecting” behavior before and after the associated method calls, providing the services enumerated above (instrumentation, error handling, etc).

Using DataServices, you could define methods like this:

[source:c#] [Log()] [RequiresRole(Role.Admin)] [Database(ConnectionType.SQLServer)] public bool CreateRecord([RegExp("[a-z]*”)] string name, [Range(1, 99)] int age) { // just open the connection and insert, no further checking needed! } [/source]

As you can see the CreateRecord method contains .NET attributes that define the valid ranges of execution for the parameters, and has some other attributes that define pre- and post-conditions to be checked prior to execution. This allowed us to centrally manage a quite large framework (~20 classes, ~100 quite complex methods) and centralize the debugging, security, logging and parameter checking routines into a single location.

Moreover, using configuration files, one was able to “inject” (“weave”, using AOP terminology) more behaviors into this mechanism without having to recompile the application (this is known nowadays as “Dependency Injection” or “Inversion of Control” – Fowler, 2004), while the core “facade” methods only contained specific instructions related to the business rules that they implemented. The code was lighter, easier to read and maintain, and largely self-documenting.

It is worth noting, however, that we used a very small subset of the functionality that AOP provides, that is, the runtime interception of messages sent to objects and weaving aspects before and after method execution:

“Another way to understand AOP is in terms of how it works. A common misconception associated with this perspective is to equate all of AOP with just one part of the supporting mechanisms. This kind of error is analogous to saying that OOP is just abstract data types. Probably the most common mechanism error is to equate AOP with interceptors.(…) It’s true that AOP does use functionality like interceptors and Lisp advise. It also incorporates techniques from reflection, multiple inheritance, multi-methods and others. However, AOP has an explicit focus on crosscutting structure and the modularization of crosscutting concerns. The rule of thumb to avoid mechanism errors? Remember that AOP is more than any one mechanism—it’s an approach to modularizing crosscutting concerns that’s supported by a variety of mechanisms, including pointcuts, advice and introduction.”

(Kiczales, 2004)

It is also interesting to note some drawbacks about our approach:

1. Performance: the runtime-based interceptions of method calls was heavy, adding a 10-factor more processing time at each method execution. However, since our system was not designed for high number of users or transactions per second, this was not a showstopper, at least not in that phase of the project.
2. Debugging: the Visual Studio .NET 2002 debugger (used at the beginning of the project) had some trouble figuring out the “jumps” between the “normal” methods and the AOP code (which we called “hyperspace” in our jargon, since the call stack of the weaved code seemed to appear and disappear completely, almost magically, while debugging). The workaround was to set up breakpoints in strategic places around the code.
3. Microsoft support: while .NET provides basic AOP capabilities, such as message interception and context-bound objects, these features were undocumented and explicitly non-supported; however, the code created back in 2002 still compiles perfectly well in later versions of the .NET Framework.

Conclusion

AOP helped us to have an extremely high productive environment, and in only 4 months we had a working system (4 people were working full-time in the project), providing an enourmous amount of functionality, and enhancing the current system.

Unfortunately, though, the project could not be finished completely, since the venture capital group that financed the company cut all financing unexpectedly and all development tasks stopped in 2003. Nevertheless, it was a highly rewarding experience, that gave me a practical insight into complex OOP, AOP and architectural design, as well as into project management techniques.

References

Fowler, Martin, “Inversion of Control Containers and the Dependency Injection pattern” [Internet] http://www.martinfowler.com/articles/injection.html (Last accessed July 2nd, 2006)

Hilsdale, Erik; Kiczales, Gregor et al., “Aspect-Oriented Programming with AspectJ™”, Xerox Corporation, http://www.ccs.neu.edu/research/demeter/course/w03/lectures/lecAspectJ-w03.ppt (Last accessed July 2nd, 2006)

Jacobson, Ivar; Ng, Pan-Wei, “Aspect-Oriented Software Development with Use Cases”, ISBN 0-321-26888-1, Addison-Wesley, 2005.

Kiczales, G., Lamping, J., Mendhekar, A., Maeda, C., Videira Lopes, C., Loingtier, J.-M., and Irwin, J, “Aspect Oriented Programming”, Springer-Verlag, 1997, [Internet] http://www.cs.ubc.ca/~gregor/papers/kiczales-ECOOP1997-AOP.pdf (Last accessed July 2nd, 2006)

Kiczales, Gregor, “Common Misconceptions”, Dr. Dobb’s Magazine, February 10th, 2004, [Internet] http://www.ddj.com/showArticle.jhtml?articleID=184415113 (Last accessed July 2nd, 2006)

Shukla, Dharma; Fell, Simon; and Sells, Chris, “Aspect-Oriented Programming Enables Better Code Encapsulation and Reuse”, MSDN Magazine, March 2002 [Internet] http://msdn.microsoft.com/msdnmag/issues/02/03/aop/ (Last accessed July 2nd, 2006)

Let’s say that you have a base class, called, well, BaseClass (the examples are in C++, but you can take this idea to any other OO language):

[source:c]

class BaseClass { public: // constructor, destructor, copy // constructor, assignment operator…

virtual void doSomething();

}

void BaseClass::doSomething() { // provide some basic behavior here, // and a comment that says something like:

// IMPORTANT NOTE to implementors of subclasses:
// this method should always be called before your code!

}

[/source]

And then you derive a class from this base class, called, well, DerivedClass:

[source:c]

class DerivedClass : public BaseClass { public: // constructor, destructor, copy // constructor, assignment operator…

virtual void doSomething();

}

void DerivedClass::doSomething() { BaseClass::doSomething(); // WTF???

// do something else...

}

[/source]

As stupid as it might sound, this code is hard to understand, debug and maintain, because it includes an unneeded dependency from the child to the base class – that is, another one, besides the fact that one derives from the other!

The problem is that the “doSomething()” message appears here as actually two messages: one which is mandatory and generic (like initialization or verification), and the other, a specialized one (which is why people use polymorphism, actually). One is public, the other is not. Mixing both messages creates a semantic problem in the above code, which ultimately makes the code harder to understand.

This might lead a myriad of other small things; what if another developer (just landed in the team), months later has to add a new child class of BaseClass, and forgots to (or just does not know that she should) call the BaseClass::doSomething() method? This might cause pain, either because it generates a bug that’s hard to track, either because the developer spends time trying to understand why the new class does not work while she’s working on it. In any case, it is a cost that you, as an architect, might have avoided in the first place. And if your designs are poorly documented (which is a sad truth), or if the developer that created the code has gone away from the team (which happens a heckuva lot of times every day), well, your chances of losing time and effort in your project will grow accordingly.

In my opinion, such an approach goes against the very idea of polymorphism, and I prefer to refactor such a code this way:

[source:c]

class BaseClass { public: // constructor, destructor, copy // constructor, assignment operator…

void doSomething();

protected: // a pure virtual, abstract method // without implementation, working as a // “hook” for subclass implementors virtual void doSomethingElse() = 0; }

void BaseClass::doSomething() { // provide some mandatory, basic behavior here

// hook for subclasses to extend this behavior!
doSomethingElse();

}

[/source]

And then:

[source:c]

class DerivedClass : public BaseClass { public: // constructor, destructor, copy // constructor, assignment operator…

protected: // provide some extended behavior! virtual void doSomethingElse(); }

void DerivedClass::doSomethingElse() { // do something else… }

[/source]

Ahhh… I much more prefer this way of doing things:

1. First of all, it makes BaseClass an abstract entity, an interface, a pure thought creation, a divine entity above all the earthling vacuum; you want to program against interfaces, and not be tempted to use them as concrete things. That’s what subclasses are for.
2. The call to doSomething() will always, always (let me repeat that, always) be called, no matter how sloppy the creator of DerivedClass is. The worst thing you can have is an empty DerivedClass::doSomethingElse() implementation. In that case, no harm.
3. If the developer is so, so, so sloppy that it forgets to implement DerivedClass::doSomethingElse(), well, guess what: at some point in time, either the compiler or the linker will complain! And this is a good thing (at least in languages that feature a compiler, of course!)
4. You’ve unknowingly created a nice API, by the way. You can document this with a good UML diagram, and it will stand out in your Doxygen or JavaDoc documentation.
5. It’s easy to understand and hard to break, as all good designs should be.
6. Your developers will love your work, and your karma gets a bonus point. The next life you might as well approach a higher level of consciousness and save Mankind from its horrible destiny. (OK, maybe this is too much, but hey, who knows?)
7. The approach is called the “Template Design Pattern” and this will add another bonus, but this time for your CV ;)

Hope this helps! Any comments welcome, as usual.

The question is: would it be useful if in hardware, each cell of data would carry its own type designation? I will discuss here the pros and cons of this approach, in respect to hardware and software architectures.

Introduction

The example of the + sign is particularly interesting; here’s an excerpt of a tutorial for the Ruby programming language, where the need for data types appears in a straightforward way:

“Before we get any further, we should make sure we understand the difference between numbers and digits. 12 is a number, but ’12′ is a string of two digits. Let’s play around with this for a while: puts 12 + 12 puts ’12′ + ’12′ puts ’12 + 12′ (results) 24 1212 12 + 12 How about this: puts 2 * 5 puts ’2′ * 5 puts ’2 * 5′ (results) 10 22222 2 * 5″

(Chris Pine, 2006)

As we can see in the above example, higher-level programming languages allow us to distinguish (if not explicitly like Java, contextually like Ruby) among different types of information, whereas, at hardware level, this distinction does not exist; the processor executes or processes instructions or data depending on the processor cycle.

Metadata

In other words, if meaningful information (data) is stored in the memory of the computer and can be processed by a computer program, then we can say that every bit (no pun intended) of information has, at least at a certain abstraction level, and from a certain point of view, a particular type, or, more generally, some metadata attached to it:

“Metadata (Greek: meta- + Latin: data “information”), literally “data about data”, is information that describes another set of data. A common example is a library catalog card, which contains data about the contents and location of a book: It is data about the data in the book referred to by the card. Other common contents of metadata include the source or author of the described dataset, how it should be accessed, and its limitations.”

(Wikipedia, 2006)

In our case, the type of a particular piece of data is part of its metadata:

“Assigning datatypes (“typing”) has the basic purpose of giving some semantic meaning to otherwise meaningless collections of bits.”

(Wikipedia, 2006)

(Imaginary) Type-checking processor architecture

Now, let’s suppose that a certain processor architecture allows us to distinguish in-memory pieces of data from in-memory program instructions. How could this be implemented?

First of all, let’s see the different primitive types of information that a processor could distinguish:

• Integer numbers (of different but defined lengths such as 8, 16, 32 or 64 bits)
• Floating-point numbers (again, of different but defined lengths)
• Single characters (single- or multibyte-characters, such as Unicode ones)
• Strings (of variable lengths)
• Pure binary streams (images, audio, video)
• Uniform arrays or vectors (of variable length, where the items are all of the same type)
• Variable arrays or vectors (of variable length, where the individual items can be of any type, similar to C structures)

Let’s imagine, to begin, that this is the definitive list of supported types at hardware level, by a certain microprocessor architecture. I have kept this list particularly close to that of any common high-level language, for reasons that will become obvious in a while.

How could the type metadata be stored at hardware level? The easiest way to imagine this is having a supplemental byte at the beginning of each in-memory variable or structure, indicating the type. This would give us the possibility of referencing 256 different types of data, which is more than enough in this particular example.

In the case of variable length data types as shown above (String, Arrays) another byte (or bytes) should indicate the length of the whole data structure. The need for this will be explained below.

Type-checking

Now, during the execution type, the processor would fetch data from memory but this time, it would have a first byte of information about the information (metadata) indicating the type of what follows, and eventually some length information as well. Instead of having to rely in context (which is the case by now), the processor could proactively check that the data will be processed by the appropriate instructions. This is a common technique used in programming languages called “Type Checking”:

“The process of verifying and enforcing the constraints of types – type checking – may occur either at compile-time (a static check) or run-time (a dynamic check). Static type-checking becomes a primary task of the semantic analysis carried out by a compiler. If a language enforces type rules strongly (that is, generally allowing only those automatic type conversions which do not lose information), one can refer to the process as strongly typed, if not, as weakly typed.”

(Wikipedia, 2006)

In the case of a processor-based type check, it would be a pure strong, dynamic one.

Of course, this introduces the first drawback of this approach; while current processor architectures bypass this check and blindly trust the “context coherence” between data and instruction, the processor would have to execute an internal check to verify them prior to executing the instruction. A smarter approach would be to have the processor to “trust” some executing code, if it comes from a statically-typed compiler, for example; this way, the processor would not execute the type check, and would have a similar behavior as that from current systems; however, it would execute a supplemental check for code coming from dynamically-typed languages (such as scripting languages).

Security

A direct benefit of type-checking processor architectures such as the one described above has to do with security. One of the most common security problems in software today is inherent to the Von Neumann architecture, in which data and instructions are both loaded in memory and share adjacent locations. This security problem is known as “Buffer Overrun” or “Stack Overrun”:

“A stack-based buffer overrun occurs when a buffer declared on the stack is overwritten by copying data larger than the buffer. Variables declared on the stack are located next to the return address for the function’s caller. The usual culprit is unchecked user input passed to a function such as strcpy, and the result is that the return address for the function gets overwritten by an address chosen by the attacker. In a normal attack, the attacker can get a program with a buffer overrun to do something he considers useful, such as binding a command shell to the port of their choice”.

(Howard & LeBlanc, 2003, page 129)

(Howard & LeBlanc follow this statement with a C program that shows the security failure, and explain how it might be exploited to inject code in the computer program and change its behavior)

In the case of the stack buffer overrun, the problem is not only that current processors do not check the type of the data to process, but they do not even check the length of it. Length-checks should be then the first check that a processor should do before processing in-memory data of variable length (as stated above, Strings, Arrays and structures like C structs fall in this category).

This leads to infere that a type-checker processor would be particularly useful in security intensive environments (such as nuclear plants, life support systems, etc) where the tradeoff of performance for an additional security type check could be highly desirable.

Virtual Machines

Virtual machines such as Java or .NET’s Common Language Runtime (CLR) both perform length and type checks; in those cases, the virtual machine specification ensure that the code being executed does not perform illegal memory accesses or reference objects of the wrong type at the wrong moment.

For example, .NET’s CLR includes a runtime security engine that constantly checks code metadata, to know whether the method calls are trusted or not:

“Permission demands propagate up the stack. When a method call demands a particular type of permission, the security engine must affirm that every component in the stack (prior to the point of the permission demand) has appropriate permissions. If any component does not, the permission demand fails and an exception is thrown to signify this failure. Each frame of the stack can modify the effective set of permissions by calling Assert, Deny or PermitOnly before making calls, and there are also calls to Revert changes made earlier. Taken together, this mechanism results in aggregate behavior that is constrained by the least privileged component that is participating in a given stack region”

(Stutz, Neward & Shilling, 2003, page 185)

This, among other reasons (such as automatic memory management) make virtual machines a “hot topic” in computing nowadays, since they allow to develop much more secure systems, with greater productivity, with fewer resources.

Of course, it must be said, not all security problems have disappeared with virtual machines, but that’s another topic.

Other benefits

I think that another performance benefit could come from the fact of having native string manipulation at hardware level. String manipulation is by far the most common operation performed in high level programming languages (where Perl and Basic are the most common examples), but yet until now text strings as such exist only at that high level (and even until recently, when the Standard Type Library appeared, C++ did not even have a native string type – http://www.bgsu.edu/departments/compsci/docs/string.html).

Common string operations that could be implemented at hardware level include string copying, string concatenation and splitting; this way, getting the length or a defined substring of a given string would need a single processor instruction, instead of the current procedures, that imply memory allocation and copying, both extremely expensive in time and resources:

“In all cases, these functions consist of copying all or a subset of a string to another string. The specific steps are:

1. Determine the number of characters to copy
2. Allocate space for the characters
3. Copy the characters to the new string

Because of the memory allocation and copying operations involved, extracting sub-strings is also an expensive operation.”

(VBIP.com, 2006)

Conclusion

The inclusion of high-level instructions in processors is not something new. It allows to boost the speed of hardware architectures, providing common operations to be performed at maximum speed at the lowest system level. One good example of existing implementations is the Velocity Engine existing in PowerPC G4 and G5 microprocessors:

“The Velocity Engine, embodied in the G4 and G5 processors, expands the current PowerPC architecture through addition of a 128-bit vector execution unit that operates concurrently with existing integer and floating-point units. This provides for highly parallel operations, allowing for simultaneous execution of up to 16 operations in a single clock cycle. This new approach expands the processor’s capabilities to concurrently address high-bandwidth data processing (such as streaming video) and the algorithmic intensive computations which today are handled off-chip by other devices, such as graphics, audio, and modem functions.The AltiVec instruction set allows operation on multiple bits within the 128-bit wide registers. This combination of new instructions, operation in parallel on multiple bits, and wider registers, provide speed enhancements of up to 30x on operations that are common in media processing”

(Apple Computer, 2006)

Some example code in C that uses the AltiVec instruction set is shown in http://developer.apple.com/hardware/ve/tutorial.html

Of course, these implementations greatly impact compiler and operating systems design, but they do not (I think) impact higher-level languages such as Java, C#, or scripting languages such as Perl or Ruby, who tend to be rather platform-independent (both software and hardware).

References

Apple Computer, “Velocity Engine” [Internet], http://developer.apple.com/hardware/ve/ (Accessed February 3rd, 2006)

Chris Pine, “Learn to Program” [Internet], http://pine.fm/LearnToProgram/?Chapter=02 (Accessed February 3rd, 2006)

David Stutz, Ted Neward & Geoff Shilling, “Shared Source CLI Essentials”, ISBN 0-596-00351-X, O’Reilly, 2003

Michael Howard & David LeBlanc, “Writing Secure Code, 2nd Edition”, ISBN 0-7356-1722-8, Microsoft Press, 2003

VBIP.com, “String Operations” [Internet], http://www.vbip.com/books/1861007302/chapter_7302_04.asp (Accessed February 3rd, 2006)

Wikipedia, “Datatype” [Internet], http://en.wikipedia.org/wiki/Datatype (Accessed February 3rd, 2006)

Wikipedia, “Metadata” [Internet], http://en.wikipedia.org/wiki/Metadata (Accessed February 3rd, 2006)