This isn't meant to be grief, and I suspect this sentiment isn't unique to you. I wish to address the point without pointing any criticism at you.
I understand what you are saying - the responsibility falls on someone else: the person who left the litter. The reality is that the litter is there and the responsible person is unlikely to return to pick it up and no one else is hired to do that task. Given that reality, the sentiment of 'not my job' - while accurate - is unsustainable. There is a paradox inherent in wanting a shared resource to be maintained but being unwilling to put in effort to make it so. I suspect anyone who has lived in a roommate situation can attest. For instance, I lived in a three bedroom apartment where a dirty salad dish sat in the sink for two weeks. All other dishes were washed and dried but that one was left because everyone thought someone else would do it. Turns out it was left by one of my guests, leaving me feeling amply foolish. This scenario plays out with cleaning bathrooms, vacuuming floors, and so on. If one person doesn't want to carry their weight, the decision falls on the other roommates to pick up the slack or live with the consequences.
I find this analogy holds when it comes to litter in the woods. In this scenario, you aren't the crappy roommate who left the dish - you're one of the other innocent roommates that faces a decision on whether or not to put in the effort to make up for the mistakes of another. While failing to put forth effort (leaving the trash/dish) isn't a detraction, it is a missed opportunity to make an improvement. If there is to be a community resource - and public trails should qualify - then a critical mass of effort is required to sustain it. If someone enjoys the trails and is capable of contributing but does nothing, then they are taking advantage of those who do contribute.
I recognize that there are more ways to contribute than just picking up litter - removing blow downs and clearing out water bars are certainly huge contributions that will do more to preserve the trail than picking up litter. But I believe there are people who utilize the trails and don't contribute towards maintenance. Do I begrudge them? No. I doubt there are many, if any, people out there maliciously enjoying the labor of others while knowingly failing to contribute. I do admit that when I read a trip report that says a trail needs major work, I wonder if that person has volunteered their time to make the improvements they wish to see. Though not perfect - there is always room for improvement - I feel fortunate that the resource we all enjoy is generally well cared for by so many.
