GNU make shell assignment patch (version 2) by David A. Wheeler, 2010-10-25

I propose adding support for the BSD "shell assignment" operator, and I've attached a patch to add this support.  This patch is "version 2"; the previous version of this patch used popen(), but this patch uses make's own shell function (and thus is even more portable).  It also has documentation to match, and more tests.

Many "make" implementations support the shell assignment operator "!=".  This includes the "make" of FreeBSD, OpenBSD, and NetBSD:
http://www.freebsd.org/cgi/man.cgi?query=make&sektion=1
http://www.openbsd.org/cgi-bin/man.cgi?query=make&apropos=0&sektion=0&manpath=OpenBSD+Current&arch=i386&format=html
http://netbsd.gw.com/cgi-bin/man-cgi?make+1+NetBSD-current
It is also supported by the "pmake" program of Adam de Boor:
http://www.freebsd.org/doc/en/books/pmake/variables.html

The semantics are not complicated.  Given:
  string1 != string2
The macro named string1 is a recursively-defined variable, with a value determined by the following process: First, string2 is *immediately* evaluated. Second, that resulting evaluation is passed to the shell for execution. Third, that result is modified removing a single trailing newline (if there is one), and then replacing all remaining newlines with spaces.

GNU make can't currently handle BSD makefiles with "!=", since it doesn't have the operator.  It's not even easy to simulate with the current GNU make constructs.  In many cases, this GNU make construct will have the same final result:
  VAR = ${system echo "$(STUFF)" | sed -e 's/://'}
but notice that $(VAR) is re-evaluated *EVERY TIME*, creating efficiency problems, and the change in result may be surprising.  GNU make's ":=" doesn't quite do it either:
  VAR := ${system echo "$(STUFF)" | sed -e 's/://'}
In this case, in GNU make it's executed only once, while with BSD make the result is recursively evaluated.  Thus, in GNU make using := you cannot insert $(VAR2) as the output of the shell script and have it be re-evaluated.  Nothing in GNU make does this exactly... so those wanting this functionality will want it added.

The following patch adds this capability to GNU make, and includes test cases.  This patch works against GNU make per its CVS repository of 2010-10-24.

I needed to modify the func_shell function slightly so I could implement the BSD semantics (which only remove the LAST newline, if any, instead of ALL the newlines).

This patch is released under either GNU GPL version 2 or later, or
the BSD 3-clause license.  Thus, it's compatible with lots of licenses
including the current and older licenses used by GNU make.

This patch is also available from:
 http://www.dwheeler.com/misc/shell-assignment.v2.patch
The older patch is available from:
 http://www.dwheeler.com/misc/shell-assignment.patch


==================================================


--- ./read.c.orig	2010-08-13 22:50:14.000000000 -0400
+++ ./read.c	2010-10-23 18:52:42.285751829 -0400
@@ -2470,7 +2470,7 @@
      w_colon        A colon
      w_dcolon       A double-colon
      w_semicolon    A semicolon
-     w_varassign    A variable assignment operator (=, :=, +=, or ?=)
+     w_varassign    A variable assignment operator (=, :=, +=, ?=, or !=)
 
    Note that this function is only used when reading certain parts of the
    makefile.  Don't use it where special rules hold sway (RHS of a variable,
@@ -2521,6 +2521,7 @@
 
     case '+':
     case '?':
+    case '!':
       if (*p == '=')
         {
           ++p;
@@ -2540,7 +2541,7 @@
 
   /* This is some non-operator word.  A word consists of the longest
      string of characters that doesn't contain whitespace, one of [:=#],
-     or [?+]=, or one of the chars in the DELIM string.  */
+     or [?+!]=, or one of the chars in the DELIM string.  */
 
   /* We start out assuming a static word; if we see a variable we'll
      adjust our assumptions then.  */
--- ./function.c.orig	2010-08-07 04:46:06.000000000 -0400
+++ ./function.c	2010-10-25 14:28:42.091107002 -0400
@@ -1396,11 +1396,11 @@
   \r  is replaced on UNIX as well. Is this desirable?
  */
 static void
-fold_newlines (char *buffer, unsigned int *length)
+fold_newlines (char *buffer, unsigned int *length, int remove_multiple_newlines)
 {
   char *dst = buffer;
   char *src = buffer;
-  char *last_nonnl = buffer -1;
+  char *last_nonnl = buffer - 1;
   src[*length] = 0;
   for (; *src != '\0'; ++src)
     {
@@ -1416,6 +1416,10 @@
 	  *dst++ = *src;
 	}
     }
+  if (!remove_multiple_newlines && (last_nonnl < (dst - 2)))
+    {
+      last_nonnl = dst - 2;
+    }
   *(++last_nonnl) = '\0';
   *length = last_nonnl - buffer;
 }
@@ -1578,12 +1582,20 @@
 #ifdef VMS
 
 /* VMS can't do $(shell ...)  */
+
+char *
+func_shell_base (char *o, char **argv, int remove_multiple_newlines)
+{
+  fprintf(stderr, "This platform does not support shell\n");
+  die(EXIT_FAILURE);
+}
+
 #define func_shell 0
 
 #else
 #ifndef _AMIGA
-static char *
-func_shell (char *o, char **argv, const char *funcname UNUSED)
+char *
+func_shell_base (char *o, char **argv, int remove_multiple_newlines)
 {
   char *batch_filename = NULL;
 
@@ -1762,7 +1774,7 @@
 	{
 	  /* The child finished normally.  Replace all newlines in its output
 	     with spaces, and put that in the variable output buffer.  */
-	  fold_newlines (buffer, &i);
+	  fold_newlines (buffer, &i, remove_multiple_newlines);
 	  o = variable_buffer_output (o, buffer, i);
 	}
 
@@ -1776,8 +1788,8 @@
 
 /* Do the Amiga version of func_shell.  */
 
-static char *
-func_shell (char *o, char **argv, const char *funcname)
+char *
+func_shell_base (char *o, char **argv, int remove_multiple_newlines)
 {
   /* Amiga can't fork nor spawn, but I can start a program with
      redirection of my choice.  However, this means that we
@@ -1854,12 +1866,18 @@
 
   Close (child_stdout);
 
-  fold_newlines (buffer, &i);
+  fold_newlines (buffer, &i, remove_multiple_newlines);
   o = variable_buffer_output (o, buffer, i);
   free (buffer);
   return o;
 }
 #endif  /* _AMIGA */
+
+char *
+func_shell (char *o, char **argv, const char *funcname UNUSED)
+{
+  return func_shell_base (o, argv, 1);
+}
 #endif  /* !VMS */
 
 #ifdef EXPERIMENTAL
--- ./tests/scripts/features/shell_assignment.orig	2010-10-23 23:14:22.797322744 -0400
+++ ./tests/scripts/features/shell_assignment	2010-10-25 16:05:33.554492781 -0400
@@ -0,0 +1,63 @@
+#                                                                    -*-perl-*-
+
+$description = "Test BSD-style shell assignments (VAR != VAL) for variables.";
+
+$details = "";
+
+# TEST 0: Basic shell assignment (!=).
+
+run_make_test('
+.POSIX:
+
+demo1!=printf \'  1   2 3\n4\n\n5 \n \n 6\n\n\n\n\'
+demo2 != printf \'7 8\n \'
+demo3 != printf \'$$(demo2)\'
+demo4 != printf \' 2 3 \n\'
+demo5 != printf \' 2 3 \n\n\'
+all: ; @echo "<$(demo1)> <$(demo2)> <$(demo3)> <$(demo4)> <${demo5}>"
+',
+              '', "<  1   2 3 4  5     6   > <7 8  > <7 8  > < 2 3 > < 2 3  >\n");
+
+# TEST 1: Handle '#' the same way as BSD make
+
+run_make_test('
+foo1!=echo bar#baz
+hash != printf \'\043\'
+foo2!= echo "bar$(hash)baz"
+
+all: ; @echo "<$(foo1)> <$(hash)> <$(foo2)>"
+',
+              '', "<bar> <#> <bar#baz>\n");
+
+# TEST 2: shell assignment variables (from !=) should be recursive.
+# Note that variables are re-evaluated later, so the shell can output
+# a value like $(XYZZY) as part of !=.  The $(XYZZY) will be EVALUATED
+# when the value containing it is evaluated.  On the negative side, this
+# means if you don't want this, you need to escape dollar signs as $$.
+# On the positive side, it means that shell programs can output macros
+# that are then evaluated as they are traditionally evaluated.. and that
+# you can use traditional macro evaluation semantics to implement !=.
+
+run_make_test('
+XYZZY = fiddle-dee-dee
+dollar = $$
+VAR3 != printf \'%s\' \'$(dollar)(XYZZY)\'
+
+all: ; @echo "<$(VAR3)>"
+',
+              '', "<fiddle-dee-dee>\n");
+
+
+# TEST 3: Overrides invoke shell anyway; they just don't store the result
+# in a way that is visible.
+
+run_make_test('
+
+override != echo abc > ,abc ; cat ,abc
+
+all: ; @echo "<$(override)>" ; cat ,abc
+',
+              'override=xyz', "<xyz>\nabc\n");
+
+
+1;
--- ./variable.c.orig	2010-08-27 11:01:42.000000000 -0400
+++ ./variable.c	2010-10-25 15:47:07.292252038 -0400
@@ -1111,6 +1111,30 @@
   return var;
 }
 
+/* Given a string, shell-execute it and return a malloc'ed string of the
+ * result. This removes only ONE newline (if any) at the end, for maximum
+ * compatibility with the *BSD makes.  If it fails, returns NULL. */
+
+char *
+shell_result (const char *p)
+{
+  char *buf;
+  unsigned int len;
+  char *args[2];
+  char *result;
+
+  install_variable_buffer (&buf, &len);
+
+  args[0] = (char *) p;
+  args[1] = NULL;
+  variable_buffer_output (
+    func_shell_base(variable_buffer, args, 0), "\0", 1);
+  result = strdup(variable_buffer);
+
+  restore_variable_buffer (buf, len);
+  return result;
+}
+
 /* Given a variable, a value, and a flavor, define the variable.
    See the try_variable_definition() function for details on the parameters. */
 
@@ -1120,7 +1144,9 @@
                         enum variable_flavor flavor, int target_var)
 {
   const char *p;
+  char *q;
   char *alloc_value = NULL;
+  char *alloc_value2 = NULL;
   struct variable *v;
   int append = 0;
   int conditional = 0;
@@ -1140,6 +1166,13 @@
          target-specific variable.  */
       p = alloc_value = allocated_variable_expand (value);
       break;
+    case f_shell:
+      /* A shell definition "var != value".  Expand value, pass it to
+         the shell, and store the result in recursively-expanded var. */
+      q = alloc_value = allocated_variable_expand (value);
+      p = alloc_value2 = shell_result (q);
+      flavor = f_recursive;
+      break;
     case f_conditional:
       /* A conditional variable definition "var ?= value".
          The value is set IFF the variable is not defined yet. */
@@ -1357,6 +1390,8 @@
 
   if (alloc_value)
     free (alloc_value);
+  if (alloc_value2)
+    free (alloc_value2);
 
   return v->special ? set_special_var (v) : v;
 }
@@ -1432,7 +1467,7 @@
 	  return (char *)p;
 	}
 
-      /* Match assignment variants (:=, +=, ?=)  */
+      /* Match assignment variants (:=, +=, ?=, !=)  */
       if (*p == '=')
         {
           switch (c)
@@ -1446,6 +1481,9 @@
               case '?':
                 *flavor = f_conditional;
                 break;
+              case '!':
+                *flavor = f_shell;
+                break;
               default:
                 /* If we skipped whitespace, non-assignments means no var.  */
                 if (wspace)
--- ./doc/make.texi.orig	2010-08-29 19:05:27.000000000 -0400
+++ ./doc/make.texi	2010-10-25 23:07:14.353440174 -0400
@@ -1379,6 +1379,7 @@
 @cindex =, expansion
 @cindex ?=, expansion
 @cindex +=, expansion
+@cindex !=, expansion
 @cindex define, expansion
 
 Variable definitions are parsed as follows:
@@ -1388,6 +1389,7 @@
 @var{immediate} ?= @var{deferred}
 @var{immediate} := @var{immediate}
 @var{immediate} += @var{deferred} or @var{immediate}
+@var{immediate} != @var{immediate}
 
 define @var{immediate}
   @var{deferred}
@@ -1408,12 +1410,21 @@
 define @var{immediate} +=
   @var{deferred} or @var{immediate}
 endef
+
+define @var{immediate} !=
+  @var{immediate}
+endef
 @end example
 
 For the append operator, @samp{+=}, the right-hand side is considered
 immediate if the variable was previously set as a simple variable
 (@samp{:=}), and deferred otherwise.
 
+For the shell assignment operator, @samp{!=}, the right-hand side is
+evaluated immediately and handed to the shell.  The result is stored in the
+variable named on the left, but that variable becomes a simple variable
+(and will thus be re-evaluated on each reference).
+
 @subheading Conditional Directives
 @cindex ifdef, expansion
 @cindex ifeq, expansion
@@ -5400,6 +5411,7 @@
 @cindex =
 @cindex :=
 @cindex ?=
+@cindex !=
 
 To set a variable from the makefile, write a line starting with the
 variable name followed by @samp{=} or @samp{:=}.  Whatever follows the
@@ -5454,6 +5466,37 @@
 endif
 @end example
 
+One way to execute a program and set a variable its result
+is by using the shell assignment operator @samp{!=}.
+This operator, which is compatible with various BSD make implementations,
+first evaluates the right-hand side, then passes that result
+to the shell for execution.
+If the result of the execution ends in a newline, that one newline is removed;
+all other newlines are replaced by spaces.
+The resulting string is then placed into the named
+recursively-expanded variable.
+If the result of the execution could produce a '$', and
+you don't intend what follows that to be interpreted
+as a make variable or function reference, then replace every '$'
+with '$$' as part of the execution.
+For example:
+
+@example
+hash != printf '\043'
+file_list != find . -name '*.c'
+@end example
+
+You can also set a variable to the result of running a program using
+@code{shell} function calls (a GNU make extension).
+@xref{Shell Function, , The @code{shell} Function}.
+For example:
+
+@example
+hash := $(shell printf '\043')
+var := $(shell find . -name "*.c")
+@end example
+
+
 @node Appending, Override Directive, Setting, Using Variables
 @section Appending More Text to Variables
 @cindex +=
--- ./AUTHORS.orig	2010-07-12 21:20:10.000000000 -0400
+++ ./AUTHORS	2010-10-25 22:13:27.600440201 -0400
@@ -61,6 +61,7 @@
   Carl Staelin (Princeton University)
   Ian Stewartson (Data Logic Limited)
   Ramon Garcia Fernandez <ramon.garcia.f@gmail.com>
+  David A. Wheeler <dwheeler@dwheeler.com>
 
 With suggestions/comments/bug reports from a cast of ... well ...
 hundreds, anyway :)
--- ./variable.h.orig	2010-10-23 11:08:36.036623200 -0400
+++ ./variable.h	2010-10-25 11:21:15.978999570 -0400
@@ -135,6 +135,8 @@
                            const char *replace, const char *pattern_percent,
                            const char *replace_percent);
 char *patsubst_expand (char *o, const char *text, char *pattern, char *replace);
+char *func_shell_base (char *o, char **argv, int remove_multiple_newlines);
+
 
 /* expand.c */
 char *recursively_expand_for_file (struct variable *v, struct file *file);

