I think it's not moving by using XML, Because It's commenting in module-catalog/templates/product/list.phtml file.

    <?php echo $block->getToolbarHtml() ?>