“Why do you advise plunging right into a programming project instead of carefully planning it first?

If you’re trying to solve a simple, predefined problem like doing a depth-first search, thinking everything out beforehand doesn’t hurt. But few real problems are like that. In real-world applications, you don’t usually know at first precisely what problem you’re trying to solve. So if you spend a lot of time planning in advance, what you’ll end up with is a minutely detailed plan for solving the wrong problem.

With complex, ill-defined problems, you’re better off writing a prototype as fast as you can, seeing what turns out to be wrong with it, and then changing your definition of the problem accordingly.

Often the reason programmers are pushed into planning is not that the problem requires it, but that project managers require it. Maybe programmers should give managers an explicit choice: do you want me to solve the problem in the way that will make you feel good, or the way that will yield the best solution.”
Paul Graham – Programming FAQ

To give you an example, I’m currently trying to start making a calculator (with GUI) which manages to calculate physical data, such as force based on the electromagnetic force between two objects and such. In order to do that, I could startup by sorting everything. I would most likely try to make some code which may calculate the electronic force or the gravitational force, all depending on the input values. That is, because these are similar. And here you see the issue with planning: I can’t make prototypes fast if I start with optimalization fast. Thus, I can’t check for errors, nor enjoy the prototype itself.

Now, instead of messing all that stuff up, I’ll try and make a function which focus on special relativity instead. What about the momentum? Let’s start off with that:

We know that

and that c = 3.0 * 10^8 m/s

A simple function which calculates the momentum would thus be written in Java as

static final double c = 3.0 * Math.pow(10, 8);

static momentum(double m, double v) {
    double p = m * v / gamma(v);
    return p;
}

static gamma(double v){
   return (Math.sqrt(1 - Math.pow(v/c, 2));
}

(I’m writing in Java because Applets and Lisp doesn’t seem to be friends at the current moment.)

Most likely, I would have put the whole expression into the momentum-function in the beginning. That is, I would put the gamma-function directly into the momentum-function. After I notice that the momentum-function works fine this way, I drag it out and recheck.

Most people know this already, but I just want to ensure people know it and do it. Also, please avoid thinking about optimalization at the current moment, you’ll have tons of time for that later on. Besides, it’s more fun to actually be done, then optimalizate later on.

x = 5
newLst = [y[x] for y in [lst, array, ht, string]]

(setf x 5
      new-lst
  (list (nth x lst)
        (aref array x)
        (gethash x ht)
        (schar string x)))

Now, as shown, python is incredibly much shorter due to its ability to get data from lists, arrays, hash-tables and strings. Are we able to recreate this with macros in Common LISP? Possibly, yes.
The main issue is just the hash-table, which you cannot use the function elt on. Otherwise, you could just make dispatch-macros and use elt accordingly with it. So this macro has to evaluate and check the type of the element we’re dragging out. Consider it as a cond-evaluation (I prefer if-ing it):

(if (typep ,sequence 'hash-table)
    (gethash ,@args ,sequence)
(if (typep ,sequence 'array)
    (aref ,sequence ,@args)
    (elt ,sequence ,@args)))

The “prettiest” way of using this macro is most likely by making the macro operate in this way:

[hash-table key]
[multidimensional-array 1 2 3]
[list 4]

Now, we can define the former long expression to a rather short one:

(setf x 5
      new-lst (mapcar #'(lambda (y) [y x])
              '(lst array ht string)))
; Or  (loop for y in '(lst array ht string) collect [y x])

Albeit, it’s rather inefficient compared to the former expression.
But why stop there? I mean, in Python, I used the [x:y:z]-syntax to make a simple PNG-parser which gave the pixel-value of each pixel in the image. Why shouldn’t Lispers have this possibility as well? To be honest, there is none but the lack of such an idea.

>> x = “Hello World!”
>> x[:5] # Evaluates to x[0:5]
'Hello'
>> x[::2] # Evaluates to x[0:len(x):2]
'HloWrd'

In lisp, this would be possible to express as

(setf x “Hello World!”)
[x :5]
[x ::2]

Now, expanding it to have more pythonic ways of working, we may add negative numbers as well:

[x -1] ; Last character in the string
[x :-1] ; All but last character in the string
[x ::-1] ; Reverses the string.

I would assume this would shorten lisp-code by a lot, due to the increase of abstraction. But, as we all know, this comes with a price: Slower speed.

I’ve thought a lot about that issue, but I came to the conclusion that with a good compiler, this would only be slower in cases where you haven’t defined the type of the elements. And those cases usually seem to be a prototype-stadium.

Another thing which I would really want a compiler to do when we’re at it, is making code more efficient by converting the first statement into the second statement

(write-string [string 0:5] out) ; Inefficient
(write-string string out :start 0 :end 5) ; Efficient

This also yields for all other functions which works like this one.

But let me go deeper. I want a code which exceeds Python’s capacity in terms of abstraction. In order to do that, I’m going to state something which is obvious for most people in the programming world: Data within a computer is based upon bits: zeroes and ones.

Now, what has this to do with sequences and hash-tables? Pretty much everything: A number is a number, but is also an array of bits. A string is a string, and an array of characters. A character is a converted number, and a number is an array of bits. What about manipulating these bits through Lisp?
In order to actually manipulate bits and bytes in Lisp through this method, you’ll have to think multidimensional arrays: All elements are made up either by arrays of bits, arrays of arrays, or a combination. We could say that all elements are made up by ts and nils, apart from t and nil.

In order to manipulate this data directly would mean that setf-ing this would be a destructive modification. Not only that, but in order to speed up the code, a []-array would return pointers to the bits, bytes or object. Also, the array would have an element-type (unless we’re bracketing a hash-table or an array)  to prevent possible errors. In other words, this would always return a displaced array. How to get a copy instead? Take advantage of copy-seq or copy-alist.
An example of how this could be utilized is by making a xor-function which repeat an xor-pattern on an ASCII-string:

(defun xor (xor size string)
  "(xor xor-num size string) returns an xored-string where xor-num is
   the number which will perform the xoring, and size is the amount
   of bits the xor-number is (e.g. 32 or 64, though ‘non-byte’
                                   values may be used as well)"
   (let ((xor-bitlst (coerce
                       (copy-seq [xor :size])
                       ‘list))
         (new-string (copy-seq string)))
     (nconc xor-bitlst xor-bitlst) ; Circular list.
     (loop for xor-bit in xor-bitlst
           for val-bit in [new-string : :8]
           for i
           do (setf [new-string : :8] (logxor xor-bit val-bit)))
     new-string))

Which raises a lot of interesting questions: How do one actually represent an integer as an array if bits? One may assume that [x] would return the “whole” bit-array of x. But for an integer, [x] would be infinite, so that’s out of the question. Rather, [x] would be short for [x :(1+ (integer-length x))]. Thus, [1] would return #*10, -1 would yield #*11, -4 would yield #*001. Basically, the bit-vector returned would be signed, and in the “opposite” way.

In order to find out whether a number is negative, one can just do [x -1]. If [x -1] returns true, then the number is negative. Else, the number is zero or positive. Trying to modify the value of this would give you an error.
Take note that [x y:z] will always give an error if x is an integer, y is a non-negative integer and z is a negative integer.
If y is a negative integer, then the array returned could have been represented as (make-array (-  y z) :element-type 'bit :initial-element [x -1]). However, as this array would be impossible to modify, it would be causing more trouble than usefullness.

Another interesting thing included, though most likely not frequently used, would be the offset-addon. Python has the list[x:y:z]-possibility. Lisp should have the [list w:x:y:z]-possibility, where y is the offset for the list. This would work as :displaced-offset-index in make-array. Thus, (eq [list :::4] [list :4]) should be true, and (subseq list (+ w z) (+ x z)) would eq [list w:x::z].

Now, only the last part remains: A macro which will be implemented in any lisp-compiler which actually does this job for us. It will give programs a faster development-process, and it would be about as fast as programs without these macros as well. It’s a win-win situation.