A Crisis of Numbers

Here is a silly problem for which no simple solution exists, as far as I know. My inability to make sense of it makes me question my own judgment so much that I'm less sure than usual that I even exist.

Suppose two envelopes are placed in front of you, and you are truthfully told that one of the envelopes has twice as much money in it as the other one, but you don't know which one is which. One of the envelopes is randomly assigned to you. You look in the envelope and discover that it contains X dollars. You are given the opportunity to trade in this envelope for the other one before you walk off with your money.

Suppose you are given an envelope in this manner every day. On average, do you earn more money by switching or by sticking with the envelope originally assigned to you?

Conventional wisdom says that it is pointless to swap envelopes as long as you have no idea which one contains more money. But then explain what is wrong with the following logical illogic: You know that there is a 50% chance that the other envelope contains 1/2 of X dollars, and there is a 50% chance that the other envelope contains 2X dollars. Therefore, if you swap envelopes, you're "average expected outcome," or "expected value," is equal to 50% of 2X plus 50% of 1/2 of X, for a total of X and 1/4 of X, which is more than X.

This is known on wikipedia as "the two envelope problem."