Often you will come to situation where you want to show specific content in a different way for mobile and desktop users. Sure you can use some CSS trickery of displaying or hiding some portion of the content but the fact is that hiding is not the same as not loading at all. It will affect the speed and on mobile devices issue of page loading speed is even more important than on the desktop.
What can a mortal do to speed up a WordPress website? A lot can be done but if you are stuck on a slow server you should definitely consider paying few extra bucks for a better hosting service.
So is it easier to code your own design? Of course it is. I stepped in to the world of design some 15 years ago and in to coding a little bit later. Since i do both things I can really tell the difference between coding of my own design and coding the client’s design.