"There is no incentive in this game for being playing nice in Dark and darker."

Why should there be? as you point out yourself...

"You're actively hurting yourself and your team as there will be more blue portals to fight over."

That is correct. and also there will be less loot. and less XP gained. and a greater chance of getting backstabbed/betrayed.

"It felt pretty bad killing new groups in normal lobbies."

....why? you don't need to feel bad lol. it's a video game. you aren't hurting anyone. if a new player dies, they just re-join a new game in 30 seconds.

"I would also say that there is even less mercy in high roller due to scores for loot and kills."

Of course there is. again, why should there be mercy at all? this isn't a wow pve server. pvp combat is an intrinsic part of this game. for a lot of us, it's the whole point of the game.

In this game, the pve is braindead. it's not remotely a challenge, even in high roller. the environment is easy to survive... if it weren't for the pvp, this game wouldn't be anything special. and i know i'm not the only one who feels this way.

"Should there be an incentive for being nice to other players."

That's the thing, the incentives in a game like this should be intrinsic to you and the experience you want to curate for yourself.
If you want to learn more about Dark And Darker,please vist https://www.mmoexp.com/Dark-and-darker/Gold.html