I plan to undertake a large project within the next 30 days. I will be dealing with around 8 Million simple products. I think my best plan is to work out the infastructure I'm using, and address any bottlenecks within the stack. Once I recieve the products, I plan to spin up a small Mage instance to see what kinds of issues I run into, then create the final Mage install on a larger server.
I plan to base this on Magento 1.9 - having used Magento 2 I feel it's not ready for such a project as I don't want to have to deal with issues relating to an upcoming platform.
The products won't have images for the time being, but in the near future this will be a needed expansion. My reasoning for this is I will be listing products on large marketplaces (Amazon, Ebay...etc) and will not be running a storefront for the time being.
My main areas for concern is DB load due to the amount of products, PHP being single threaded, and CPU IOPS. I know I can offset some of this using Redis as a database store, caching, SSDs...etc. I am able to utilise an hourly stock feed, which will only include products which require an update, however with that many products I'd imagine there is a lot of change in stock - which means the stock feed will still be large.
I'm leaning towards using AWS, due to it's scalability and ability to split up different sections of my setup, depending on what needs more resources. I haven't had a chance to use AWS infastructure before, but from reading documentation from RDS and S3 buckets, it looks like something that would tick most of the boxes.
If you were in my position, would your go-to be AWS or another provider? If you would use AWS, would you use S3/RDS and why?