Because they're not really yours.
Your dreams are seeds that God planted in your heart and in your soul, to start with. Turning your back on them is, to start with, an insult to the Godhead.
Not a good idea. Insult anyone else, but not Him.
Secondly, your dreams may not be entirely about, or for you. They may be intended to reach past your life to touch, and inspire others.
The film Invincible is a good example of this. It's the story of Vince Papale, who at the age of 30 went to an open tryout for the Philadelphia Eagles football team, in 1976. He had played one year of high school ball, went to a college without a football team, and played league football while working as a teacher and bartender.
And he made the team, playing for three and a half seasons before injury forced his retirement.
But it was more than his dream. Philadelphia was treated poorly by the 1970s, and football was an important binder for the community. But the Eagles were a disaster, one of the worst teams in the NFL.
But things changed in 1976, when they brought an 'old man' with a fiery heart and a big dream on board. The fans had one of their own with whom they could identify, and the team had a seemingly bottomless reservoir of enthusiasm in one player.
In 1981, the Eagles won the Super Bowl. Papale was no longer playing, but that victory is widely regarded as his legacy.
One man's dream brought pride, unity, and, yes, hope to a city of millions.
(The movie is well worth seeing...produced by the Disney studios, it's a great family-friendly film.)