Add to all grids
If you want to add your custom export format to all grids at once, like it is done with the default xml and csv exports, then have a look at the following GitHub issue, at the moment there is a bug to accomplish this: https://github.com/magento/magento2/issues/24543#issuecomment-530265065
Update Feb 2021
The issue will not be fixed by Magento. So it is just not possible.
Add to specific grid
If you want to add only to a specific grid (e.g. order grid) , then use the following:
Add file {vendor} /{module} /view/adminhtml/ui_component/sales_order_grid.xml
with the following content to your module:
<?xml version="1.0" encoding="UTF-8"?>
<listing xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Ui:etc/ui_configuration.xsd">
<listingToolbar name="listing_top">
<exportButton name="export_button">
<argument name="data" xsi:type="array">
<item name="config" xsi:type="array">
<item name="options" xsi:type="array">
<item name="custom_csv" xsi:type="array">
<item name="value" xsi:type="string">custom_csv</item>
<item name="label" xsi:type="string" translate="true">Custom CSV</item>
<item name="url" xsi:type="string">{your_frontname}/export/gridToCustomCsv</item>
</item>
</item>
</item>
</argument>
</exportButton>
</listingToolbar>
</listing>
Now create a controller to handle the request.
definition.xml
to your custom module from this path: >vendor\module\view\base\ui-component\etc\definition.xml