So rather than moving each button, I ended up adding a second Wishlist button manually in Magento_Catalog/templates/product/list.phtml
like so:
<div data-role="add-to-links" class="actions-secondary"<?= strpos($pos, $viewMode . '-secondary') ? $position : '' ?>>
<?php if ($this->helper('Magento\Wishlist\Helper\Data')->isAllow()): ?>
<a href="#"
class="action towishlist"
title="<?php echo $block->escapeHtml(__('Add to Wish List')); ?>"
aria-label="<?php echo $block->escapeHtml(__('Add to Wish List')); ?>"
data-post='<?php /* @escapeNotVerified */ echo $block->getAddToWishlistParams($_product); ?>'
data-action="add-to-wishlist"
role="button">
<span><?php /* @escapeNotVerified */ echo __('Add to Wish List') ?></span>
</a>
<?php endif; ?>
</div>
Then I removed the original Wishlist button:
<referenceBlock name="category.product.addto.wishlist" remove="true" />