You need to consider serious modifications in your site if you feel your website belongs to those which has lower performance but impressive design. For calculating the genuine performance of your website with user prospect, you need to look insight into your web analytic account and need to identify the matrices which show user behavior and help you take any decision which adds value to your website performance. Below are some ideal guideline for building website which performs well with both prospects.
Do you have flash, animation, video or any other rich media elements which can take longer time, make sure that they have been integrated proper way. Also evaluate whether your visitors really show any interest in viewing those elements or not. Some analytic tools provide in page web analytic feature which you can use to see the percentage of clicks on those elements.
Type of website which is built for providing good content, there is no need to have creating fancy design. People want to have clear readable content which can be accessed from a mobile and different browser with same experience. So make sure, you have equal visual effect in different environment.
Displaying thousands of products distributed in different categories, it may possibly that visitors directly search for specific products of which he is looking for rather than having a look at large navigation menu. Make sure you have sufficient search option in your website which works properly.
Error free coding is essential for website performance, make sure CSS and HTML have optimal errors.