When I was last out canvassing for Labour, the message I got from a lot of voters was: 'I won't vote - they're all the same'. And they had a point: even under New Labour, inequality actually increased. They engaged in some positive tinkering around the edges, but there was no substantive change to the status quo.
When the only real mechanism for significantly improving working people's lives has been airbrushed out, then all they're left with is simplistic, blame it on the so-and-sos (i.e. anyone but the real beneficiaries of the system) propaganda. It's the so-and-sos (Europeans, immigrants, benefits claimants, black people, jews, socialists, Russians, etc.) that are to blame for their increasingly shitty lives.
IF Benn is right (but is he?) and you keep voting Labour no matter how far right they go (btw. Starmer literally just said there's now no significant difference between Labour and the Tories on immigration policy), you are contributing to this rightwards drift.
The answer to the anguished question 'Why is this happening?!' would be 'You are making it happen'.