I've had several conversations about a project with a coworker about a project that is well underway - 3 out of 4 weeks for this particular portion of the app have gone by. The problem we are facing is that the code is not in a workable state yet and the effort put into fixing the bugs has produced several design changes and re-codes of large chunks already. Specifically, there are problems in a recursion portion of the app - only the first 2 levels of the recursion are working. Beyond those 2 levels the nodes start acting like they are being added as a child of the root instead of being added to the appropriate level as a child.
A recent conversation we had concerning this went something to the effect of:
Coworker says:
argh :@ this makes no freaking sense
I say:
one of the things i like to do in those situations is start over in a scratch project... write the most basic simple code that you can, to achieve something in the direction that you are heading. once you get it working in the scratch project, go back to the real project and re-implement whatever you need, to make it work like the scratch project. also - welcome to the world of recursion. :) if you aren't familiar with the concept of recursion, i would highly recommend that you start with some simple small recursive method examples - a method that calls itself. the trick is figuring out how to stop the recursion so that it does not become an infinite loops.
Coworker says:
but im not getting an infinite loop, and if i thought the project was simple enough to start from scratch i would be for that idea, also dont think i am going to have time to start from scratch
I say:
i'm not suggesting you rebuil the whole thing from scratch. i'm suggesting you create a junk prototype project that you can throw away, and implement the absolute most basic form of what you are doing. no databse connections or anything hta trequires external resources. just create a control that loads itself in a recursive manner a given number of times, and allows you to fire some event from the control and catch that event from the appropriate control level. wrap your head around the most simple form of what you are trying to do, so that you have a better understanding of how the real project should work. once you've done that, going back to the real project should be easier.
as far as time is concerned... look at it this way: you've spent the last 12 + hours trying to figure this out. if you spend the next 12 hours no getting it fixed, that's 24 hours (3 days) of no progress. if you spend the 12 hours writing a simple example of what you are tryign to do, and you fully understand the pitfalls after that, then you have only wasted 12 hours and have better utilized the other 12 hours. the total effect of either way is that there are 24 hours of time used that has not made progress on the app. the benefit of spending the 12 hours learning is that after the total of 24 hours has been used, you are better equiped with the knowledge you need to fix the problem. instead of just banging your head against the keyboard for 24 hours hoping something will work.
hindsight is 20/20, as the saying goes.... now that you are so far down this path you can see that the largest issue you are having is the recursion of sub-questions. with that being known, it is a very good idea to learn all you can about recursion, within a reasonable ammount of time. mitigate the risk that see by examining that risk in the most basic form possible. then you can apply your new knowledge to the original problem and have a much better chance of seeing where the problem is
The point is, don't be afraid to take a step back and prototype a process. It's important to understand that prototyping can be a very beneficial process and can actually lead to time saved. The process of learning and working through problems before applying knowledge to a production system will only lead to better quality code and faster development timelines in the production code. If you spend 12 hours banging your head against a wall of a problem and you know it will take you another 12 hours to figure out how to fix the problem, then you are likely to waste a total of 24 hours of your time. If however, you realize that you can spend the next 12 hours creating a simple prototype that teaches you where the problems are in the current code, then that same 24 hours will produce a net effect in favor of your development effort. At the end of scenario a, you have wasted 24 hours and you might have finally figured out how to fix the specific issue but you haven't actually shown that your "fix" will solve the issue. At the end of scenario b, you have explored the reasons that the issue exists and what the options are for working around the issue. You have not implemented the issue yet, but you understand why the issue is what it is and you also understand how to fix the issue, knowing that your fix will solve the problem. Which one sounds like a better use of your time?
What to prototype
The big question is "what do you prototype?" - unfortunately there is no easy answer to this. It always comes down to your specific scenarios. As a general guideline, any identified area of risk within a system is a prime candidate for prototyping before writing the final version of the software. The tough part is understanding what is a risk. Every project has it's own unique areas of risk and every developer also has their own areas of risk. An example of a project risk may be the operating system version that is being targeted for the production environment. Is this software supposed to run on a PC? If so, will it be Linux or Windows? If Windows, will it NT4, Vista, or whatever other version? This level of risk has to be addressed at the outset of the project's requirements gathering and is not likely to be a good candidate for prototyping unless you know that your software is supposed to support mutliple platforms (Linux and Windows. PC vs. Handheld. etc). There are also risks of what specific 3rd party software will be utilized. Will our message queue system be MSMQ, MQSeries, or some home grown system? This level of risk applies not only to the architectural decisions made during the project requirements gathering but also in the daily lives of developers writing code. This risk may need some level of prototyping to ensure that the developers and admins of the system know how to run the chosen queue system. However, this risk may not need code prototyping. It is far more common (and more appropriate, IMO) for this risk to be mitigated through the use of Dependency Injection in your software. There are plenty of other blog posts that cover this area of risk mitigation and I won't go into the details here.
The more likely scenario of a project that needs code prototyping from a code standpoint actually has more to do with the developers on the project. Every developer has their own unique experience set with their own comfort zone in development. For example, I am very comfortable writing ASP.NET applications that dynamcially add controls to the screen at runtime in a recursive manner. However, my coworker from the above conversation has not had any experience with that scenario. Unfortunately, I made a few assumptions about the level of difficulty that this scenario provides - I assumed that this was not an area of risk due to my own knowledge and comfort level in this scenario. Had I taken the time to sit down and ask my coworker about this task, I would have known that he did not have any prior experience with recursion or adding controls to a screen dynamically. There are now 2 unique areas of risk in this particular process based entirely on the previous experience of the developer.
The end result is that you need to prototype any area of system that is considered a risk, and the level of risk must be assesed by not only the system's architect or tech lead but also by the individual person working on the individual task. If the architect is comfortable with the technology and process then the project may not need to list this as an official risk. However, if the tasked developer is not comfortable with this area, then the tasked developer must consider this to be a risk in the project and I would highly recommend that the developer do the appropriate prototyping for that area.
When to prototype
The second big question to ask is "now that I know what I should be prototyping, when do i do it?". The easy answer is - as soon as you recognize the risk. The correct answer is somwhere close to the time that you will need to implement the real system. I've found over the years that if I do my prototype work too far in advance, I lose most of the knowledge by the time I am ready to implement the real system and I end up doing a lot of cut-n-paste from the prototype into the production code. Cut-n-paste from a prototype is not always a bad thing, but is something that should be done rarely (more on that in a moment). At the same time, you have to find a good balance between too soon and too late. Not only does your project have a timeline that needs to be met, but you also have a brain that needs time to fully encode the knowledge that you have gained from the prototype. If you try to write the prototype 30 minutes before you write the production code, chances are you won't get to the production code in time or you will be forced to scrap the prototype and move directly into the production code. If you don't give yourself enough time between the prototype and the production code, either, you will find that your brain is still stuck in prototype mode and you will write the code in the same manner as the prototype. The trick is to find the correct balance of time between writing a prototype and writing the production code. You need to write it just far enough in advance that you will remember the lessons learned, but not so far in advance that you have to resort to cut-n-paste of code because you don't remember what's going on.
How to prototype
Simplicity: The most important understanding of how to prototype is that you do not want to implement the entire system in your prototype. You only want to implement what is absolutely necessary to mitigate the original risk that was identified. If you are working with a risk that involves a very complicated set of database calls to create a recurive hierarchy of controls on a screen, cut the prototype down so that it excludes the data access - only include the recursion, use of controls in a hierarchy, and the most basic function that the controls need to have (usually raising an event so a process can run).
Iteratively: The second understanding of how to prototype. I think a good rule of thumb to rollow is the same as for writing a paper - write the first draft then shove it in a drawer for a period of time and don't think about it. Then come back and re-read the first draft after you have cleared your mind of that draft's thought process. From here, rewrite the entire paper so that you are not stuck with trying to make your original assumptions and constraints fit within your new understanding of the issues. This rewrite may involve starting from complete scratch or may only involve reorganizing your code to match your new knowledge. The choice of how to rewrite comes down to personal preference, but it is important to do a rewrite whenever possible. Does iteration always mean that you should scrap your current work and start over? If you are encountering problems that are inherent to your design (realizing that your object should actually be split into 2 objects, for example), then yes - start over with the peice that you know is designed wrong. If you are encountering problems that are inherent to the implemetation and not the design (figuring out how to add an event handler, for example) then no don't start over - just fix the problem.
Iteration in your prototype allows you to correct your mistakes as you go and form new understandings of the problem and new thoughts on how to solve the problem without encumbering you with the constraints of the original solution. Iterative doesn't apply to only the idea of starting over, though. It should be performed at every level of of the prototype including the process of building complexity into the prototype.
Accretion: The third understanding of how to prototype. The idea of software accretion in prototyping is that you start with the smallest possible part of the problem. For example, if you are working with a risk that involves recursion, the first thing you want to do is write a simple routine that performs the desired recursion. Don't worry about any of the real details of the end-goal and don't try to code the entire solution in the first pass (remember the second understanding: iterative). Code only what is absolutely neccessary for the first pass to function. Once you have coded this first pass, step up one level. If your recursion is dealing with adding a hiearchy of controls to a screen, the next step may be to add the user controls to the screen. Don't make the controls do anything yet, though. Just get them added to the screen appropriately. From there you can make the control do something - but only one thing. What is the most basic thing that this control needs to do? Perhaps it needs to fire an event when a button or is clicked - add the button click to the control and ensure that the button click works at each level of the generated hierarchy. The next step .... and on and on the accretion goes. This accretion concept should sound very familiar to anyone that has worked with Test Driven Development - the TDD process is modeled entirely on accretion.
A Prototype Is Not Production Code: The fourt understanding of how to prototype. Don't put production level error handling into your prototype and don't put production level business process into your prototype. Don't put any production level code in your prototype, period - your prototype is exactly that; a prototype. You should expect zero code reuse from your prototype to your production application. You should expect to completely rewrite the functionality that your prototype created, in the production application (remember understanding number two, again: Iterate!). You should not expt to copy-n-paste from your prototype to the production system. Some of the largest mistakes I've seen in production level applications are caused by developers who are addicted to OPP - Other People's Programming. Copy and Paste is evil in production development and only causes headaches because the simple scenario that you copied from will never be as complex or fully featured as the scenario in your production level application. If you are serious about iteration and you are serious about your prototype being a prototype, then you will take the time to rewrite the code from your prototype into your production application.
When to Stop
Having written a succesful prototype and now understanding where the pitfals are in the specified area of risk, there needs to be a stopping point for work on the prototype. The general rule to follow for when to stop working on the prototype is to ask yourself when your understanding in combination with the prototype has progressed to a level where the next logical step is to begin introducing the production system code into the prototype. If you find yourself facing a set of recursively created controls that all raise and respond to the appropriate event, and your next thought on how to continue the prototype is to add production level error handling, business logic, or process; you should stop working on the prototype.
...
Code Complete 2
Disclaimer: In Code Complete 2, Steve McConnell discusses both System Accretion (Chatper 2) and prototyping (Chapter 5) specifically. Unfortunately, this is the only readily available resource I have for both of these concepts. I have read about both of these concepts in other places at other times, however. If I have plageurized any work of Steve's in this post, then please consider this disclaimer to be my citation reference.