That is the most important thing here!
And not only should all server installation instructions have a step
called 'Creating /robots.txt' and an example with 'Disallow:
/cgi-bin/' with it, any software that creates information or scripts
that should not be indexed, like statistics packages, query frontends,
database gateways.. should come with /robots.txt and specific instructions!
Now it is just a question of who is going to do the real work here,
to list all such software with developers contact addresses, and
formulate a letter that impresses them to include these instructions
into the next release.
I believe it might even work.