ok after a few hours of searching magento 2, I found an answer for this question.
public function setStatus($orderId="") {
$orderModel = $this->_objectManager->create('\Magento\Sales\Model\Order');
$order = $orderModel->load($orderId);
$currentState = $order->getState();
$currentStatus = $order->getStatus();
$save = false;
if ($currentState !== $orderModel::STATE_NEW) {
$order->setState($orderModel::STATE_NEW);
$save = true;
}
if ($currentStatus !== $orderModel::STATUS_FRAUD) {
$order->setStatus($orderModel::STATUS_FRAUD);
$save = true;
}
if ($save) { $order->save(); }
}
This will set the payment sate to new and set the status to Pending.
can also use other status like
const STATE_PENDING_PAYMENT = 'pending_payment';
const STATE_PROCESSING = 'processing';
const STATE_COMPLETE = 'complete';
const STATE_CLOSED = 'closed';
const STATE_CANCELED = 'canceled';
const STATE_HOLDED = 'holded';
const STATE_PAYMENT_REVIEW = 'payment_review';