Thank you for your questions. There is a popular hadith often quoted in Sufi sources (although it is in neither Sunni or Shia canonical works) where it is reported that God said that He was a hidden treasure and loved to be known and so He created the creation.
Interpreters of this hadith rarely understand it as meaning that God needed to be known. God is completely needless of all of His creation and their gratitude, let alone humans. Additionally, there is no need for such a complicated system of existence for God to be known, as even the knowledge of Him on this corporeal plane is limited. Rather, His want is not out of need of appreciation, it is out of love for His creation. He wants you to know Him for your own good, so He created you to fulfill that purpose. So to understand why God created, you have to look at His mercy rather than any kind of need. It is the perfection of a human to know God and thatis what he is created for. It is intrinsic in the very fibre of his existence and leads to his ultimate happiness. God created the human for him to attain something very great, something that no other of His creation could achieve. That is out of His Mercy, Grace and Wisdom and we attain it through servitude.
Different levels of hardship are due to a variety of causes. God only tests a human to the extent of their ability. Some have greater ability than others and so their tests are more. Conversely we may not perceive something as a test when it concurs with the way we would like things to be. For example, few people may consider ample wealth as a test especially when compared to poverty. However every bounty has a corresponding responsibility and so wealth may be a harder test than poverty. But what a person should keep in mind is that life is full of tests and for a believer those tests will never be more than they can handle.
Hardship also occurs through the natural consequence of a person's bad choice through his own free will. This is a fact of life and God has placed a system of wisdom to counter act bad choices. He is All-Powerful and able to correct bad choices but it is also for a person to learn the ways of wisdom and avoid bad choice as much as possible.
As for those that commit sins and still seem to get off without hardship, their situation has to be understood with both this world and the next in mind. While they may not experience any immediate consequences in this world that is more dangerous as the consequences in the next world are more severe. This goes back to God's Wisdom.
Coming to this world is a chance and an opportunity. An opportunity is not unfair.
May Allah bless you and help you to increase yourself in every way.
'Abd Allah Esmail