Having just gone through this process myself, I understand how much of a hard decision it can be. So let's go through a few processes that I went through to know what I needed thoroughly. Needed is a key term here, as sometimes what you want, versus what you need are entirely different.
Why was I looking for a new lens?
In my personal instance, I was looking for a replacement to my current 50mm f1.4 Sigma lens. One of my favourites to use on my Canon 7d but now I am shooting on a Sony A7ii it has unfortunately taken a back seat. The main reason being that I had to have a converter and sadly, even though I didn’t buy the top of the range, I still spent decent money on the converter, yet it still cuts out, and the focus is slow.
So what 'features' does it need?
One of the main reasons I enjoyed the previous lens was because of the narrow aperture and creamy bokeh. So this would be a must in whatever I chose to buy next. The other thing that I wanted to start taking more of, partly because I have a baby boy arriving imminently, was to take portraits. In my case, I shoot full frame, which means the most suited lens for portrait work is a focal length of 85mm.
An important part of buying a lens and how to know your purchasing the right lens for your needs. The best way to do it is just writing a list, it can be precise of verbose, but until you have one, you will not be able to make an informed decision on what you are purchasing. It’s always best to tart with the basics:
What would you like to shoot with the lens?
The biggest consideration when choosing your new lens should always come from a place of what you would like to shoot. Are you planning on doing portrait shots? If so then you will probably want something around the 85mm focal length with a moderate aperture so you can separate the subject from the background with hopefully a nice bokeh at the same time.
Are you shooting wildlife? You're almost definitely going to need a zoom lens of some kind as you simply won't be able to get close enough to your subject. One thing to be aware of with zoom lenses is that they can have a variable aperture, this means when at full zoom the lens can be a lot slower (higher aperture) meaning you may get motion blur if shooting something mid-flight. If you can afford to buy fixed aperture on a zoom lens, do as it will help in the long run. Other ways to combat it is, of course, to have your camera with higher sensitivity. (ISO)
If you are doing more travel photography work, then my personal favourite is a 35mm prime. It will work in most situations. Obviously, the only caveat is you don't have any zoom. However; this can become more of a tool to make you look at framing your shots better.
Do you need it to work in low light conditions?
The answer to this question usually is yes as you want the flexibility, there are some cases where you wouldn't need to worry so much. For instance if you aren't capturing moving things, wildlife, people, etc. Landscape photography, for example, does not need to have an excellent low light capability. As in most cases, you will be on a tripod to get the shot as steady as possible. If you are trying Astro lapse photography, you may want something a little quicker, however.
One thing that will help in this situation if you can find it. Not all lens manufacturers will publish this detail, is finding the t value of the lens. The t value relates to the amount of light a lens will throw on the sensor, more commonly advertised for cine lens, however, f stops and t values can differ. For instance, in my case an F1.4 meant T1.5, but in some cases, it was T1.8 so always worth checking.
Does it need autofocus?
Something you should question when it comes to buying a new lens, if you are new to photography then sometimes autofocus is imperative. However, the flip side of that is that it will make you understand how a lens focuses and you can dial in your photographs with a manual lens.
Other things that affect this decision is the style of photography you are using this lens. If it is anything action based, then you will naturally need autofocus as the action will have already happened before you have focused and taken the shot. If you are doing landscape photography, other than it being quick, there is no need for autofocus, in fact in my experience for landscape work autofocus fails to get the focal point you are after.
How large a lens are you willing to carry with you?
Very often overlooked when purchasing your next lens. How large the lens drastically changes its portability. For instance, if you are a street photographer you don't want to be lugging around a large lens all day just to get that shot. Also, sometimes if a lens is too large it can cause you to only leave it out of the camera bag or your satchel when you leave your home. It's worth a further thought to make sure you're not buying a piece of furniture for your house.
What budget do you have?
This one is a pretty simple one; we will always want things we cannot afford. However, it's also an opportunity to be a little smarter with our purchases. Yes, you could buy that £1000 lens, but what if you could find a lens that will do as good a job for £600? That gives you £400 to buy accessories or even another lens that will give you more flexibility when shooting next.
Time to scour reviews!
Another highly important step, you will also quite often find versus studies which will take two of your shortlisted lenses and compete them against each other. Make sure you read more than one review, every photographer is different and has different needs to that bias will always come into their consideration. The best place to search for a review is simply going to Google, typing in the lens plus reviews. You will learn some of the pros, but most importantly you will get to weigh up the different cons of various items on your shortlist.
Time to try and demo some of the options
You will by no doubt have a shortlist of what you think fits the bill. The best thing you can do now is finding your local camera store and see if you can test out some of those lenses. Undoubtedly, you will have some that are over your budget, but it's worth checking what they are like so you have a comparison. Most good camera stores are quite open to you testing a lens or two but always worth phoning or emailing ahead of turning up at the store to check.
What lens did you get in the end?
So after lots of looking around considerably, I ended up getting a relatively cheap but incredible bang for the buck lens. Manual focus was something I wasn't too bothered about, but I wanted to make sure that I got a lens that let in a lot of light, so I opted for a cine lens. The Samyang 85mm T1.5. It is one hell of a lens for the price, the reason for opting for the cine lens over the photography f1.4 version is because I am planning on doing a mix of work. The long throw on the focus ring means I can be a lot more precise on the focus plane in my images. If you want to check out this lens take a look at this review.