Speaking as the owner of ~400 records (used to be over 1000!), I would say that it depends on what you like about collecting records!
When I was a teenager and in my early 20s, and records were cheaper and I went to shows every week, I think I was more likely to buy stuff. Nowadays I like having more space in my house (fewer shelves) and less stuff. So I only buy a new record if I really want it and talk myself out of everything else. And yeah records are currently very inflated. But if you just like the thrill of digging through a bargain bin or at thrift stores, and you have the space, and you love the music, go for it!
Anyway, chances are the $40 record will get resold for half the price on Discogs or in a record store in a year.