First of all, we need take a look: Log in Magento Admin > Choose a product > ADVANCED SETTINGS > Custom Options:

We can see some fields, we need to fill in these input fields.
Second, we have the APIs for updating the custom options: http://devdocs.magento.com/swagger/#resource_catalogProductCustomOptionRepositoryV1
For example:
{
"option": {
"product_sku": "string",
"option_id": 0,
"title": "string",
"type": "string",
"sort_order": 0,
"is_require": true,
"price": 0,
"price_type": "string",
"sku": "string",
"file_extension": "string",
"max_characters": 0,
"image_size_x": 0,
"image_size_y": 0,
"values": [
{
"title": "string",
"sort_order": 0,
"price": 0,
"price_type": "string",
"sku": "string",
"option_type_id": 0
}
],
"extension_attributes": {}
}
}
product_sku
: Product SKU
option_id
: We can get by /V1/products/{sku}/options
title
: Title of the custom option.
type
: Input Type: Text, File, Dropdown, Radio, etc
sort_order
: Set the order of custom attribute
is_require
: Required or not
price
: Price for the custom option
price_type
: Fixed or Percent
sku
: (updated later)
file_extension
: can be empty, allowed file types: pdf, jpg, ...
max_characters
: can be empty, used for Input type: Field, Area.
image_size_x
and image_size_y
: blank if it is not an image.
values
: can be empty, used for Select Type: Dropdown, Radio Buttons, Checkbox, Multiple Select. We need to set the title, price type, Sortorder, etc.