To change the display layout of listed product you have to create the following folder structure.
app
|- design
|- frontend
|-custom-package-name
|- custom-theme-name
|-etc
|-layout
|-template
copy the files from app/design/frontend/rwd(default package 1.9) in your custom theme.
Then edit the file app/design/frontend/custom-package-name/custom-theme-name/template/catalog/product/list.phtml.
Edit the link where color label display. And add the following condition
if($_product->getColor()) :
your html code to display the color
else :
any other logic if you want.
endif;
Help of this condition we check weather color value is associated to product or not if assign then display that color name for the product otherwise keep just display label as "Color".