home»essays

• » Resume
  (Curriculum Vitae)
• » Blog

Most Recent

• » Requiring Requirements
• » Wireless Duffel Bag
• » Exploiting the Limits of your Mind
• » Time and Pressure
• » Organizational Intelligence
• » The Good, The Bad and the Ugly of Scrum
• » Evolving Database Schemas
• » Making Java Less Difficult
• » Creating a RESTful SOA
• » Why aren't you Working on Important Problems?

Archives

Requiring Requirements

- Thursday, April 24, 2008

No one seems surprised to hear that most IT projects fail. One would think that the increasing number of delayed and over-budgeted projects would illicit some kind of existential crisis on the part of IT professionals. Many projects sustain on life support missing nearly every milestone, either to eventually die or find some inkling of success (albeit at a cost that no one would have agreed to at the start of the project). Repeating the same mistake over and over it points back to one thing: requirements. What the hell is this thing suppose to do?

Arguably, there are technical issues, people issues, and various reasons IT projects fail. There are however, necessary components to success (conversely, guaranteed failure) and all of them start with requirements. And I don't mean the kind of vague nonsense requirements espoused by analysts (e.g., "make it work with Siebel"), I mean the rigorously defined requirements that most of todays analysts and IT consultants couldn't write to save their lives.

Requirements

I have been apart of enough projects to have seen the failures and successes that are common in this industry. Examining the failures there is a common theme: lack of good requirements. Not lack of documentation, not lack of specifications or attempts at requirements. Attempts were made at requirements in almost all cases, but failures persisted in every situation where the requirements were vague, ill-defined, or unrelated to the business needs.

I have worked on such gems as "build an end-to-end data architecture", or "add live flash streaming", or "integrate at&t". These type of things always missed their deadlines and expectations. It's surprisingly difficult to meet expectations where none were set; in the IT world anything delivered in that situation will be wrong and likely your fault. Worse, if the management is poor then the lack of requirements slides downhill onto the engineers who didn't think it was their problem.

The problems can be seen a mile away. The difference between the professional software engineer and most of the developers occupying IT roles these days is that the professional engineer will not commit to any deadline without clear well-defined requirements. Any amateur can promise an "end-to-end data architecture"; but the professional will make no promises without requirements that he or she can work from.

In many cases, just the exercise of defining requirements would lead to the realization that the project isn't feasible and in some cases doesn't even need to exist. Better to know up front that your project will fail than wait months or years later, right? Not if you want to keep your job, but yes, if you want to work on successful projects.

Having well defined requirements will not ensure success, but it avoids a guaranteed failure.

Managing Success and Failure

Most IT projects like to track risks, but where is the failure tracking? Perhaps it's too negative to go around tracking how a project could fail. Call it Success Tracking so as not to offend 'the law of attraction', but a clear definition of success and a scientific approach to understanding how the project could fail would save most doomed IT projects.

Without well-defined requirements this exercise is difficult if not impossible. How do you define success or failure if the requirements are not rigorously defined? Without an understanding of success your left only with failure.

Quality Assurance

Yet another tried and tested method for killing a project is a weak or non-existent QA process. There is no panacea to software development woes, but a QA process is a necessary component to success. Further, QA is impossible without well-defined requirements. Yet in many IT organizations there are QA teams working tirelessly to assure the quality of products without any clear requirements. What kind of quality are they assuring? What does it mean to pass QA without a requirement? It's like a scientist doing a test without a theory...

Any amateur can toy with software to assure that it works the way the developer told them it works. A professional QA engineer follows the same guidance as the professional software engineer: they will not commit to assuring the quality of a product that lacks a well-defined requirement.

Required

IT projects require well-defined requirements. IT professional wishing to avoid the same repeated mistakes will formulate requirements before committing to dates. As an engineer, I am happy to work on R&D in the absence of clear requirements and with the understanding that there is no date and no committed deliverables in that phase. In most organizations this can seem an impossible feat, but this is where we separate professionals from amateurs.

REFERENCES

http://www.pbs.org/cringely/pulpit/2008/pulpit_20080418_004737.html
http://www.computerworld.com/managementtopics/management/project/story/0,10801,84266,00.html
http://www.onlamp.com/pub/a/onlamp/2006/06/20/why-do-projects-fail.html
http://www.spectrum.ieee.org/sep05/1685

by Tim Warnock at Thursday, April 24, 2008 | Post a Comment

Wireless Duffel Bag

- Friday, January 25, 2008

Wireless, adjective, having no wire; network with no physical connection
Duffel Bag, noun, a large, cylindrical bag for carrying personal belongings

If there were an impending disaster near your home and you were forced to evacuate, what would you take with you? What possessions matter most?

Looking around at your belongings, would you ask yourself why you own what you own? Do you own things that help you live your life? Do you own things that remind you of past joys? In that state of crisis when one has to determine what really matters, we quickly realize what is important.

If you live your life as I do, as so many other people live, you surround yourself with your life -- bathing in the constant reminders of past and current inspiration. Some for entertainment, some for pleasure, some for business. Your belongings often reflect the life you have lived; the outward expression of your perception of reality. In those moments where you breathe in the experience of your world, the adornments in your home shine with life.

But life is not a picture, it is not static, it continuously moves. And it does so whether you are aware of it or not.

So what happens to the possessions that once sparkled with life that now gather dust? The books once read that sit on calm shelves, never to be disturbed -- never to be experienced? We become curators to the remnants of our past; dusting the shelves of lifeless relics holding only to faint memories that these things were once important.

Do you re-read the books in your house? Do you re-watch all of the movies? Are the toys played with? And the guitars?

I found in my house a tub of Lego's and it had not been touched in years. This tub had been moved with me from place to place sort of following me around as if hoping to be used. There's a sort of sadness to Lego's going unbuilt. As a child I escaped into imaginary worlds reflected outwardly in those toy building blocks. But looking at the tub of unused Lego's, it was like a muse with no artist.

Happiness

What do you need to be happy? And what do you need to experience the most from life fulfilling your desires?

I would propose you need only two things: one is the necessities to sustain your existence, the other are the tools to live out your desires.

Necessities
The essential things to sustain your life! You need food, shelter, perhaps clothes... Think of the things you take with you when traveling.

I have realized I need clothes, toiletries, a bed, a bathroom, and food. When traveling you find the food as you go, and the astute traveler knows that a bed and bathroom can be found as needed on any journey. So you pack only the clothes and toiletries that you'll need for your journey, the rest you will acquire as you travel. And of course, the lighter you pack, the more you can do and experience while you are traveling.

In order to carry these things, the most perfect luggage is the Duffel Bag! It's light, simple, and will hold your clothes and toiletries as you move about the world. The necessities of life can fit in a Duffel Bag and go with you anywhere! And toiletries are easy to pick up as you go, so don't worry when the TSA throws them out (it's happened to me more times than I can remember).

Tools
The necessities keep you alive, but you need more to really live! This is different for everyone. For some, this might be entirely minimalistic: a pad of paper and a pen, the book you're currently reading, a laptop, a phone, etc.

What I've realized in traveling and needing to work remotely, everything that I personally need to really live can also fit in that same Duffel Bag! Given a good laptop with wireless Internet access and a phone, I can work. I can't read too many books at once so I rarely ever need more than a few books at any one point in time. Art and photography supplies have been drastically reduced as I have moved to digital.

And that giant CD and DVD collection? Pick your modern alternative (currently, I'm going with Last.fm and Netflix).

In trying to make sure all of my computers from the home to the office have access to the same files I store everything online (including version-control so I can track any edits and changes); I also do this to ensure that if one of the machines crashes then I won't lose important documents. I got to a point where all I needed was a modern computer (Mac, Windows, or Unix) and I'm easily up and running with personal and professional work. No one computer is a necessity, they are all just tools.

If this "Information Age" can do anything for you, it is to unbound you from that traditional way of life where you would need shelves full of reference materials and supplies. Unbound and wireless to live your life without being trapped in offices or studios. The information age studio can be by design, and perfectly mobile.

Live your Life

In a sort of life experiment, I decided to get rid of most of my possessions. If all I really needed to fully live and enjoy my life was so simple then I should be able to not only continue doing all of the things I do but to actually do more.

It was tricky; our minds race to think of the time and money invested into these possessions. How can we just "give them away?" Remember: the past value is not the present value -- and for possessions unused, the present value becomes a negative debt against the priceless time you have to live your life!

If you are to surround yourself with anything, it should be the in-progress life and not the past.

With that in mind I kept a bed to sleep on, a desk to work on, a camera, a computer, art supplies, and an iPhone (of course). And in this information age I have combined the computer and the entertainment center to provide me a fully integrated music, movie, television, and computing environment -- which I must admit is much nicer than running cables around the house. So I am making no sacrifice even in my laziest moments where I lounge around watching television. I can draw on my Wacom tablet staring at a cinema display in the comfort of my home or I can take it with me in my wireless Duffel Bag.

by Tim Warnock at Friday, January 25, 2008 | Post a Comment

Exploiting the Limits of your Mind

- Tuesday, December 18, 2007

Reality is an amazing thing. It bends and shapes to your perspective. Simply put, it is what you believe it is.

Where in one minute you're feeling isolated and stressed, you may simply shift the context of your thinking and feel connected and relaxed. We put our conscious attention towards something and it will become the center of our reality. As our attention is consciously controlled, so then, I argue, is our reality.

If this is the case, then the limits of our perception and our cognition could be exploited to our own ends. We invent a facade for reality that matches what our five senses tell us. My optic nerve reacts to the suns radiation as my ears react to vibrations in air. My brain takes the limited sensory data and assembles a larger picture of reality. We assume that this reality is accurate yet we forget that it is based entirely on limited sensory data with a lot of imagination.

As our brain tricks us into thinking that our perception of reality is universal (and not at all imaginary), we can very easily alter our perceptions by where we choose to direct our awareness - thus easily controlling our own reality.

The limits of ones imagination then would be the only reasonable limit to ones reality. Can you shape your waking reality with an active imagination? To what extent can we change our perceptions such that it changes our lives?

by Tim Warnock at Tuesday, December 18, 2007 | Post a Comment

Time and Pressure

- Thursday, November 15, 2007

We all drudge through life trying to survive, but why? What is so important for survival? What makes any of this life worth living? Is it a biological imperative to keep breathing for as long as possible? Or is there something we ought to be doing with our lives that motivates our survival?

The human condition seems to impress that there is a reason for our existence; a reason to survive if you will. We put ourselves in a survival mode in order to live by what we consider to be "good" or "right". Many of us never really think about what that means and instead just hope to make it through the day without stress.

A steady pressure over time has shaped humanity into creatures compelled towards survival. The same evolutionary pressure has shaped our limited capacity for reason and logic, and that limited logic tells us that there must be a reason for our survival. We plan goals and imagine the desired outcomes. We are goal-oriented to fill an evolved need for a purpose in our life. Whether we are conscious of the goals or not we carry beliefs of what our life is suppose to be...

You plod though life towards your goal and whether you like it or not that goal becomes your purpose. If the goal has not been achieved then there exists a dissonance between your reality and your purpose; a continuing source of stress that will push you towards your goal. On the other hand, if you have achieved your goal then you've also achieved your life's purpose; this can be depressing if you're so successful that you achieve all of your goals!

This goal-oriented behavior results in either stress or nihilism. I propose an alternative to the goal-oriented life that is hopefully stress-free and nihilism-free!

Time and Pressure

Given enough time and pressure we have evolved into the goal-oriented and stressful species that we are today. Within an individuals life, all of their accomplishments are the result of discreet actions towards a goal. The goal is often emergent from the day-to-day actions. Most goals change over time, evolving to better suite our needs from the actions we've taken.

I would argue that you should never be concerned with a desired outcome or goal. There is no perfect state of existence in life but there is an ideal flow that is readily achievable. Your goals are often expressions of your own ideals or virtues and they tend to change. Ironically, the closer you get to your goal the more you learn about it and hence the more the goal will change.

Imagine a river rock that is perfectly rounded and perfectly smooth. This perfectly round and perfectly smooth stone represents a goal. In our goal-oriented perspective a rough non-rounded stone would be a source of stress compelling us to chisel and polish until it's "right". But a river rock wasn't chiseled or polished; it was merely shaped over time by the pressure of the rivers water.

Many people let the water dry up and stare at rough stones planning just how round and smooth they're suppose to be and never really make any progress. If you let the water flow you'll always make progress.

Rather than focus on the specifics of your goal, focus on the time and pressure in your life and simply maintain that ideal flow. Improve with every step. Continuously refactor your life. Love every moment of the ebb and flow of this precious life. Remove focus from the goal and instead focus on the time and pressure. Make your life better, and better, and better without every worrying about what it's suppose to be. You'll end up with more accomplishments than the goal-oriented person would know what to do with.

An aside: Time and Money

Time and money are non-comparable. Money can be the result of time and pressure, but do not be so foolish as to compare time with money. Any fraction of time is worth an infinite amount of money. Or as one of my friends recently discovered: Time > $

by Tim Warnock at Thursday, November 15, 2007 | Post a Comment

Organizational Intelligence

- Saturday, August 25, 2007

Organizational intelligence is the measure of an organization's ability to comprehend and conclude knowledge relevant to its purpose. In simple terms it is the performance of a group or organization measured as a whole. For example, Basketball and Hockey require a high-degree of organizational intelligence in order for a team to win. The success of the team depends greatly on the team's ability to work together, rather than based only on individual performance.

For software development, and most business organizations, the goal of an organization is to function at the highest levels of competency across all individuals. Unfortunately, a low organizational intelligence means that an organization is only as smart as the combined incompetence of all individuals. The net effect is that simple problems are harder for the organization to solve than they would be for a single individual.

Consider the following overly simplified skill summary:




In this situation, there are three developers and a single project manager. Of the three developers, one is a database developer, one an application developer and the other a WEB/UI developer. This represents a fairly standard cross-functional team, though greatly simplified.

In order to achieve the goal this group must perform at their combined strengths. This creates a positive emergent behavior where the group dynamic is able to achieve something that no one individual on the team could perform. This is a case of positive emergence and assumes a well-functioning team with a high degree of organizational intelligence:




On the other hand, the team could be functioning in an overly competitive or chaotic manner leading to a negative emergent behavior where the group dynamic is only as competent as the combined incompetence of all individuals:




This type of scenario is fairly common in software development and engineering. Concepts such as Analysis Paralysis, Design by Committee, and Feature Creep usually result in a low organization intelligence with a negative emergent behavior.

Emergent Behavior is the result of simple entities forming more complex behaviors as a collective. The complex behavior of the group is not a property of any single entity. Group/Organization behavior is often emergent, not the fault or credit of any one individual but the dynamic formed by the group itself.

Typically smaller teams like the one in the charts perform more on the positive side than the negative. However, as teams scale the challenges quickly become less about specific skill sets but instead more about understanding and exploiting the emergent behavior of the organization in order to achieve the desired business goals.

Consider the overly simplified version:



The individuals possess the correct skills necessary to achieve the goals of the organization and ideally will surpass those goals through a positive emergent behavior of team. On the other hand, the areas of incompetence are drastic enough that given a negative emergent behavior the team will surely fail despite individual efforts to the contrary.

by Tim Warnock at Saturday, August 25, 2007 | Post a Comment

The Good, The Bad and the Ugly of Scrum

- Monday, August 13, 2007

We all hear about and we all love it: the Rugby-inspired software development methodology known as Scrum. It's fast becoming an industry buzz-word and causing many project managers to question their Gantt charts. For all the hype, what is the reality of Scrum?

Scrum is an agile-based software development methodology for project management. It is characterized by a prioritized product backlog that lists new features. Work is completed and delivered in time-boxed iterations known as sprints (e.g., two week iterations). Scrum teams are cross-functional and typically number 3-7 people each. Iterations begin with an iteration planning meeting and end with a retrospective to review what worked and what didn't.

During a sprint each scrum team gathers for a daily stand up, which is a short meeting where each person describes what they did since the previous meeting, what they're planning to do now, and any impediments. The team is self-organizing leveraging our instinctive behavior to work in small groups. The Scrum process is facilitated by a Scrum Master. That title is a bit of a misnomer since the Scrum Master carries no authority and is instead responsible for blocking any distracting influences that could disrupt the teams progress.

The principles of Scrum are well defined in the wikipedia article as well as in the book Agile Project Management with Scrum by Ken Schwaber. You can also shell out ten grand for an in-person experience with Ken. There is nothing like an expert talking about the work that you should be doing. As some of my friends and co-workers like to hear me say: Get back to work!

What's so Good about Scrum?

Delivering working software. Working software is where Scrum really shines. It's proving to be an excellent implementation of Agile Software Development with core values such as customer satisfaction and individual interaction.

There are four core values to the Agile Manifesto:

• Individuals and interactions over processes and tools


• Working software over comprehensive documentation


• Customer collaboration over contract negotiation


• Responding to change over following a plan

These are all valid principles that are easy to ignore and have proven to be hard learned lessons despite how obvious they may seem. Projects typically fail when they ignore these principles. Good documentation has never compensated for crap software. Try telling the upset customers that you gave them exactly what they contractually signed in the Service Level Agreement. The principles of the Agile Manifesto should be held as software engineering law.

Scrum provides a very effective methodology that ensures these principles through an empirical approach to software development that embraces and encourages change.

If it's so great, what's the Bad news?

No one likes to admit this, especially not Scrum advocates like myself, but Scrum fundamentally conflicts with traditional PMO. It's an interesting round of cognitive dissonance to watch a PMI-certified project manager attempt to rationalize Scrum. There are of course several ways to deal effectively with this dissonance but understand: these are fundamental differences which are not philosophically compatible.

Scrum will go far in delivering working software, but what about managing roadmaps? Better yet, what about resource allocation and God-forbid budget forecasts that are needed before a project starts? We need money and people to start a project and we'd like to know roughly how much something will cost before we agree to invest. In a perfect world we would know these answers and we wouldn't need Scrum. But Scrum exists out of necessity from the failures of so many software development projects and reminds us that the entire enterprise of software engineering is often times more like scientific discovery than building construction (which is where PMI originated, and rightfully so).

The Ugly

Scrum delivers working software in chaotic environments. At the same time Scrum is a symptom of a larger problem in software engineering such that many software projects cannot be managed like construction projects else they face increasing technical debt, unhappy customers, declining quality, going over budget, and missing deadlines.

Migrating to a Scrum methodology typically has an effect of providing early visibility to problems. The wisdom being that it's better to know that you're failing one or two months into a project rather than years. While this makes sense and this transparency is a very valuable aspect of Scrum the reality is very ugly.

Transparency to critical problems often times stems from the fundamental conflict of traditional PMO and Scrum. These are problems that are unfortunately outside of the realm of Scrum or software engineering. In this situation it is easy to attack the symptom (Scrum) than it is to address the underlying issue, which is unifying project management (roadmaps, budgets, resource allocation) with the software development.

Visibility? Be careful what you ask for! If a project is going to fail, maybe it's better to let it fail naturally than to induce ulcers in the Scrum Masters and the development staff. Remember: a good Scrum Master will be a burned out Scrum Master in most environments.

This isn't an easy problem to solve, but done wrong Scrum can create an emergent failure. Take the following anecdotal quote from Brad Wilson:
Scrummerfall. n. The practice of combining Scrum and Waterfall so as to ensure failure at a much faster rate than you had with Waterfall alone.

On the other hand: Scrum, done right, has the potential for an emergent success given iterative and continuous improvement. A potential method for solving the real problem is to exploit the emergent behavior of a system.

Emergent behaviors are difficult to track, but analyze the existing software, processes, and development and determine whether or not they are evolving appropriately. A successful development process should continually improve the same way the code itself should continually improve. The process itself should be agile, responding to change to better produce working software.

Better yet, the individuals and interactions should be agile. It is the people that must respond to change.

by Tim Warnock at Monday, August 13, 2007 | Post a Comment

Evolving Database Schemas

- Monday, January 08, 2007

Master your Domain

It amazes me that while database schema evolution is one of the most critical factors in software development it's also one of the most ignored and least understood aspects. Pretty much every interesting software development project requires an evolving database schema. From renaming tables to modifying relationships it's as critical as any piece of source code but is often the least cared for part of any project. Your schema is what maintains order in your data, arguably this is the most important part of any enterprise (organizing your data)!

There are mountains of excellent (and practical) resources on how to manage software development projects. I can get certified in SCRUM or nearly any Agile-based development methodology but I can barely find any useful websites on managing my database in an Agile environment. When your development process encourages change this applies to your data modelling and schema design and not just application development.

There are some excellent articles from Fowler and Ambler that provide a good starting point. However, I found a definite lack of details and practical advice in those articles. It goes without saying that I want to run regression tests and version my work, but a database is distinctly different in that I have to deal with all that existing data and can't exactly redeploy a schema while preserving the old data (not easily anyhow).

Existing persistence strategies often fail to accommodate Agile-based development leading to poor design choices in data modeling. In this article I'd like to explore practical solutions to Agile database development. Like anything Agile, there's no perfect solution, but this is a pervasive problem across all interesting software development projects and I hope the discussion alone will yield better solutions.

To start, there is one important observation that I've found to be true across various software development projects: If the code smells it's likely that the database smells worse! Let's examine some common database smells:

• Inconsistent relationship strategies; when your ER diagram starts looking like spaghetti and every piece of business logic introduces a different convention you've got a problem. You have object tables and three possible types of relationships (1-1, 1-N, N-M), your data model is only as complicated as you make it; pick a strategy for each of the three types of relationships and stick to it. I liken this to using GOTO statements in software, it's unacceptable.


• Inconsistent object model strategies; this is often the impetus to change your relationship strategies, that is, when I have inconsistent strategies for object tables it leads to very confusing relationships. You'll see one table with a varchar(20) NAME and another with a varchar(16) NAME, is this the name of the object or does the object contain a "name"? Use a consistent strategy for tables as well as concepts like status, timestamp, IDs and alternate keys (such as name).


• Inconsistent naming conventions; STAT_DATE, STATUS_DT, or STATDATE? Pick a convention and stick with it!


• Inconsistent usage of the same column; a common example is a varchar column named TYPE that means different things to different applications. I've noticed that even good data modellers make this mistake.


• Overloading fields; things like a comma-separated list of values where only the application knows what each value means. Don't use a relational database if this is how you model - text files may work better!


• F normal form; Johnny just took a class on database design and learned about normalizing a database and now you have 175 tables in what he claims is 6NF! There are appropriate times to denormalize just as often as there are to normalize.


• Know when to OLAP; why are there summary tables attached to each of my transaction tables?


• The Cauldron of Data; this is the crux of the problem, everyone is so scared of the data that they lose control of the schema and treat it like a bubbling cauldron too paranoid to make any significant changes out of fear of breaking a legacy application. This is the end result in any application where they didn't manage their evolving database schema.

Let's talk about some solutions!

First of all, this is a developer problem! Don't expect your DBA or Hibernate to fix this for you - if you're a developer this is your problem. This leads to the central theme of how I propose database evolution to be solved: Your development methodology must cover application and database development.

If your application depends on a database, then your development methodology better cover both application and database development! I know, you like building the code and leaving the responsibility of the database to someone else. But that brings you back to the Cauldron of Data scenario where you can't make any significant changes to a schema because it got our of your control. And if you can't control the schema you can hardly control the application that depends on that schema!

We tend to ignore database development as a way to simplify our application development - I suggest you make the application development suffer by adhering to a development methodology that works with database development! Think of it like this: you're going to be the databases bitch if you don't take this responsibility.

I know, this seems like more work from the application side but like anything done right it's hard to imagine doing it differently once you get your development methodology to cover applications and databases. That said, how does one integrate their database development into a unified development methodology?

Let's look at the differences between application development and database development (and what needs to change in the traditional Agile-based development methodologies):

• Databases contain data that cannot be lost; this means you have to migrate production rather than reinstall


• Data is easy to migrate when your data is not controlling you (see the Cauldron above)


• Rebuilding your database is like compiling and deploying your code (this sounds like a maven target)


• Databases should have unit tests, and not just for the stored procedures (more on this later)


• Map your database development to your project lifecycle goals exactly like you would with application development (say, in Maven 2) but introduce the migrate step in the deploy target.

If I compile and build my application why not build the database schemas at the same time, just like I would with anyother dependent artifact? So, let's get practical and talk about things you can actually do to accomplish Agile-based evolutionary database design:

Create a Database Schema Change Policy

Keep it simple and make sure you answer how you plan to address schema migrations planned and unplanned. Your process should lend itself to an emergent property of better schema design. This by itself requires you to not only support planned and unplanned schema changes, but to encourage them. Either do a big design up front (not-agile) or encourage change in all aspects of your development (including your data model). I recommend you clearly define a process for planned migrations (migrating from one version of the schema to another) and unplanned patches (critical fixes, the kind you get in the middle of the night).

Bring DBAs in Early

You'll need their help, and you know it, best to get friendly with them early on - give them a chance to know what you're trying to do on their database. I argue that you'll find more resistance to agile development from software developers than you will from DBAs. Most DBAs have been on the front-line fixing smelly database code and are likely your strongest ally. Not only can they help with the development process they can (and should) assist with design.

Use Stored Procedures

Read up (separately) on "End-to-End Architecture", if your schema is going to change then you better clearly define your endpoints and provide an API-like package to abstract the schema completely. What's great about stored procedures is that they can (and should) be treated like application code. I recommend two types of packages, consider using a suffix of _PKG and _API. All of your object tables will likely have GET, PUT, and DELETE procedures. These should be autogenerated, if not, write yourself a script or invest into some software to autogenerate CRUDL stored procedures. Each schema should have a _PKG with CRUDL procedures for all object tables. You also have business logic, from complicated transactions to simple procedures like authenticate(user, pass). Procedures that encapsulate business logic should be in packages with an _API suffix and follow the same rigorous design that would be employed for any application API.

The naming convention of an _API and _PKG suffix is unimportant (any convention here would be fine), but what is important is distinguishing between your CRUDL procedures and your APIs that encapsulate your business logic. Once you have a convention that cleanly separates these concepts you now have a mechanism which can completely abstract your schema from your application and best of all, you've likely imposed some constraints and standardization on your object tables that lend themselves to easy autogeneration of the CRUDL procedures.

Version your Schema just like you would an Application

Versioning is a given for application code, why should database code be any different? Schemas, default data, packages, grants, everything should be versioned along with ALL other application code. Applications depend on a versioned database, just like any other versioned artifact - I would expect my build to fail if the dependent database for my application doesn't exist.

Each of your schemas is like an application, and all of the DDLs should be checked into source control and managed as applications! Check in your test data (sql inserts) and you'll easily be able to define a database-specific unit test environment!

Use the Right Tools

You'll need more than a modeling tool, modeling tools are great at helping you to visualize your schema, but don't get carried away. You need to track schema AND default data! Use tools that fit your process not the other way around - write your own scripts if you need, they're not that hard once you have a working process. Between Maven and some sqlplus scripts we've gotten plenty of mileage at my current job with the following scripts:

• drop_objects.sql; loops through all of the schemas and drops everything, there's also a delete user approach but with the drop script you don't have to redefine your tablespace; this script is never run in production


• create_objects.sql; loops through all of the schemas and creates all of the tables and default data; this script is never run in production


• create_pkg_spec.sql, create_pkg_body.sql; loops through all schemas and compiles the package specs and separately the package bodies


• run_tests.sql; loops through all schemas and runs database unit tests, it uses stored functions with setUp and tearDown procedures similar to Junit; this script is never run in production


• migrate_objects.sql; loops through all objects and runs a per-schema migrate script which is created based on the delta between two different versions of the same schema

Localhost Development

Why else do we have fancy development workstations? Stop assuming Eclipse is allowed to eat up all of your resources - let Oracle do it! The only way to empower your developers to be agile is to give them an environment where they can easily change the database schema!

We've gone so far at my current job to support localhost Oracle instances where we checked Oracle into our software version control (along with Tomcat, Java, etc). We tried using the Express Edition but it didn't support all of the PL/SQL code we were developing so we've got the full bloated 10g running on all of the developer workstations (takes about 30 minutes to install on a new workstation). So don't tell me you can't run MySQL locally!!

REFERENCES

http://www.martinfowler.com/articles/evodb.html
http://www.agiledata.org/essays/databaseRefactoring.html
http://www.agiledata.org/essays/databaseRefactoringSmells.html
http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf

by Tim Warnock at Monday, January 08, 2007 | Post a Comment

0 Comments:

Post a Comment

Making Java Less Difficult

- Wednesday, May 03, 2006

I'm sure that you've heard this before, but I'll say it anyway: Why do Java developers make simple problems difficult? There are plenty of people who have written on that topic, and I have no desire to further expound the point. I think we all get it. What I do want to talk about is how to make things less difficult, in particular how to make web application development as easy in Java as it is in Perl, PHP, Python, or Ruby.

I challenge that the answer is not to build Yet Another Crappy Web Framework (tm). I for one, am terribly sick of complicated frameworks that are more work to configure than a web application is to create manually.

What I've noticed, is that developing in Perl is often faster than Java mostly due to the built-in regular expression handling Perl offers; and this is despite the fact Java offers a comparable regular expression engine.

Two things that slow me down as developer:

1) Staring at code that involves a StringTokenizer, while loops, and several nested substring statements where I'm counting characters on my figners. Why do Java programmers continue to write code like this? I suspect the reason is because of the next issue.

2) Wanting to use a regular expression, you first create a Pattern object, followed by a Matcher object, remember to compile your pattern, and then finally match the pattern using the Matcher object. Your matched object can use the group method, which is implemented from the MatchResult interface. I'm going to repeat myself, as this bears repeating, why do Java programmers make simple problems difficult?

These two issues cover 90% of the cause for Java development to be slower than Perl. Solving these two problems should then speed up my development in Java to be on par with my development in Perl!

Let's look at some sample Perl code:

if (/a(.*)b/) {
  print "$1 is between a and b";
}

We could do this fairly easily in Java, but let's be honest, the real strength of Perl is when I start doing things like:

 @bar = split(/:/, $_);
 @foo = grep(/^#/, @bar);
 print join(':', @foo)

While that example is simple to any Perl programmer, it is, admittedly less intuitive and less clean than a Java version. Although it is only three lines rather than the twenty or more it would take in Java. But what about something like this:

XYZ bar = XYZ.split(":", arg);
XYZ foo = bar.match("^#");
System.out.print( foo.join(":") );

This seems easy, and perhaps clean enough not to bewilder the Java developer the way Perl code tends to. Perhaps we can create a utility Java class that supports the needed data structure and methods to write code like that. What other methods would we need to empower Java developers to stop writing the usual tokenizer/substring mess? Consider just the basics:

  join - join the values of an array into one string
  split - split a string into parts
  match - match the parts
  matchAll - match repeated patterns
  trim - trim the ends of a string

This much is already provided in Java in one class or another. The String class already lets me match and split, but it's not enough. I need more than just a String class, I want an array of Strings. No, I want a dynamically sized array of Strings that I can easily get and put from. While I'm at it, I want it to be associative array, where I can treat it like a normal array and it will retain index order, or I can add my own keys (similar to PHP arrays). Given a data structure like that, add in the above methods to that data structure, and I think we'd be in business.

The scripting languages provide a default context that includes regular expressions, hashtables, dynamically sized arrays and various string manipulation features. I can do all of that in Java, but it's not conveniant creating a different object for multiple data structures and methods, especially since they're almost always part of the same context.

Let's create one class that contains the data structure and the methods. This should solve both of my issues, making it easy to do what once was difficult and speeding up development for most all applications.

Even better, we can leverage what Java is good at to get this done. Let's extend a Java Hashtable which solves most of the data structure part.

public class Preg extends Hashtable {

 public Object get( int inInt ) throws NullPointerException {
  Integer cast = new Integer(inInt);
  return this.get(cast);
 }

 public void put( Object value ) {
  Integer key = new Integer( this.size() );
  this.put( key, value);
 }
}

I've added the ability to put values without providing a key, this will allow us to use this object as both a HashTable and a Vector (i.e. a resizable array). For those of you who are upset that this is inefficient, you're absolutely right. Although I would argue that it's a negligible difference. But since we're speeding up development, this will more than compensate since I'll actually have time to optimize my code and do some refactoring before the deadline (a luxury I rarely see Java developers having time for).

The next step is to create the methods, we should also provide static versions of the methods to gain maximum convenience. For example, I may only implement one "join" method, but I'll provide several prototypes:

 public String join() {
  return Preg.join("", this);
 }
 public String join( String glue ) {
  return Preg.join(glue, this);
 }
 public static String join( Preg src ) {
  return Preg.join("", src);
 }
 public static String join( String glue, Preg src ) {
  ... the actual implementation
 }

After I do this for all of the methods, I'm left with a very powerful object that allows me to accomplish the previous Perl example:

Preg bar = Preg.split(":", arg);
Preg foo = bar.match("^#");
System.out.print( foo.join(":") );

Three lines of Java that for once is equal to three lines of Perl!

All the fancy trickery that we do in Perl or PHP can be done in Java with just the right mashup of HashTable, Pattern, and Matcher. You can split a string into parts, filter, trim, and arrange to your hearts desire without ever having to count a character offset for a substring!!


Note: I have all of this working for one of my projects, which will hopefully move as fast as it would in Perl or PHP (it's a web application, so I suspect this will in fact compensate for that 90% slowdown I mentioned earlier). If there's interest, I can provide the source and/or Javadoc.

Even with my extremely verbose Javadoc it's surprisingly not that much code.

by Tim Warnock at Wednesday, May 03, 2006 | Post a Comment

3 Comments:

• By  Tim Warnock, at Wednesday, May 03, 2006  

  For reference, this is from one of my unit tests:
  
  Preg tokens = Preg.split("/", "/some/aabsolutely/arbitrary/path");
  Preg startWithA = tokens.match("^a");
  String aPath = startWithA.join(" ");
  String output = aPath.ltrim("a");
  assertEquals( "absolutely arbitrary", output );

• By  Anonymous, at Thursday, May 04, 2006  

  Oh, and it's particularly useful in a Data Access Object (DAO):
  
  Preg results = db.exec("select 'Hello World' as HL from dual");
  //---
  assertEquals( "HL", results.preg(0).get(1) );
  assertEquals( "Hello World", results.preg(1).get("HL") );

• By  Kevin Murphy, at Monday, April 02, 2007  

  I'd be interested in your code. Whoo boy, I hear you about Java programmers making everyone's lives more difficult because they don't know what they're missing from the scripting languages. BTW, which Crappy Web Framework do you dislike the least? (;-)

Post a Comment