A Signature Resolution

Ripard Teg over at Jester's Trek has made a couple of quite spurious posts recently. It started fairly innocently, with him stating in passing that a Ferox sniping with Spike effectively has worse tracking then a Naga sniping at the same range with faction ammo.

While it's true that the Ferox's tracking speed is worse, a number of respected PVPers commented that because the Naga's larger guns have a much higher signature resolution, it's tracking in practice is actually much worse. Ripard proceeded to double down on his mistake, claiming that signature resolution is not working as intended, and that it has little effect on the ability of large guns to hit small targets. He even published an entire second post to establishing his position that, while he knew that was how signature resolution should work, it was quite definitely not working in that fashion.


While Ripard occasionally takes up controversial positions, his posts have a reputation for being well informed and well reasoned - it's quite rare to find one that's clearly and demonstrably wrong, much less so two within the space of 24 hours! I'm not generally one for internet arguments, however Ripard is currently a shoe in for a spot on CSM8, and personally I like my CSM members to be well informed about how the game works. With that in mind, I decided to jump on the test server and prove Ripard wrong.

In Ripard's own words "The math is failing... And trust me, I know math".

Well Ripard you may know math(s), but I know Science!


How Should Signature Resolution Work?

I've written about this before and even made a video about it, but here's a brief run-through.

When you shoot something, two numbers are generated:

Hit Quality - This is a randomly generated number between 0 and 1, and determines how much damage your hit could potentially do. Tracking, signature, etc have no effect at all on this number - it's random.

Hit Chance - This is also a number between 0 and 1, but isn't random - it's generated via the scary equation below. This is where things like choice of guns, angular velocity, and so on actually come into play.

Shamelessly stolen from the Agony wiki

Increasing any number on the top of that formula will decrease your hit chance. Increasing any number on the bottom will increase your hit chance (ignore the range bit on the right, that's a little different but we're not worrying about it right now).

Once the game has worked out your hit chance two numbers are compared, and if the Hit Quality is less than or equal to the Hit Chance, you hit. That's an important bit - you can tell, because it's in bold.

Hit Quality will range from 50% to 150% of your turret's base damage (aside from critical 'wrecking' shots which do 300%, and effectively occur on a roll of 0). If Hit Chance is 1 (which happens when any of those numbers on the top of the equation are 0), then every roll of Hit Quality will automatically hit. If for example it's 0.5 on the other hand, only the bottom half of all possible rolls - 50% to 100% damage (plus wrecking hits) will hit - everything else will miss.

What this means is that while things like tracking and signature resolution do not directly mean that your hits do less damage - it's still the same random roll - your good quality hits will simply count as misses instead. Note two things however - firstly, that even a roll of just over 0 still does 50% of your base damage (the reason you get tiny scratches sometimes is because multiple turrets are involved, and most of them missed). Also, note that because wrecking hits occur on a roll of 0, it is always possible to get a wrecking hit - it's just unlikely.


What Does Ripard Think?

I'm not actually sure what Ripard is claiming. I actually think he's completely misunderstood what signature resolution does - at one point he mentions that signature resolution would have an effect if both ships were stationary (which is patently false - if both ships are stationary your hit chance is perfect even with guns the size of a planet) but because people are MWDing around it becomes irrelevant. Huh.

Either way, there are a few of things that he does state quite clearly that I believe we can put to the test:

"And yet [the Naga] stomps the crap out of a Ferox, doing more than double the DPS of the combat BC... and its tracking is more than 50% better."

"See above. Sigres doesn't enter into it if the Ferox is MWDing around, which it surely will be."

"In actual practice, signature isn't much of an issue because both sides will tend to be MWD'ing around. I agree that if both ships are static, the sigres of the guns will take its toll, but in an actual battle? Not so much."

"The math is failing."

"Trust me, I understand this subject just fine." 

 
From these statements, I'm going to guess that Ripard holds the following premises to be true. If he doesn't, he's free to correct me:

• Premise 1 - A Naga with 425mms rails and faction ammo should out-track a Ferox with 250mms rails and spike by a significant margin.
• Premise 2 - Signature resolution should not have a significant impact on a ship which is MWDing around.
• Premise 3 - Signature resolution should have a significant impact if both ships are stationary.
• Premise 4 - The math of the tracking formula is not functioning as intended.
• Premise 5 - Ripard understands how signature resolution works.

Let's test them.


The Test

To prove that Ripard's claim is false, I'm going to jump on the test server and fit up a couple of ships as per his specifications. You can see the key stats for each below.

The first two are straight from Ripard's post, except that I'm using regular ammo instead of faction (the tracking is identical). Notice that the tracking speed on the 250mms is significantly lower thanks to the tracking penalty on Spike.

The third option I added as an attempt to remove tracking speed from the equation completely. With two tracking-scripted TCs, the 250mm rails with Spike have almost identical tracking to the 425mm rails. This means that any difference we see between options 1 and 3 is down almost entirely to signature resolution.

For my target, I chose a Cormorant with shield extenders and a MWD. With everything running, the Corm's signature radius comes up to a massive 551m - larger than the signature resolution of 425mm Railguns. Surely if there's a situation where signature resolution is irrelevant, this is it.


Establishing a Baseline

First of all, I tested each weapon system with both my shooter and my target stationary. I will test each once with the Cormorant's MWD running, and once without. This should give me a baseline to compare against, and if I'm correct we should see the following:

• Because both ships are stationary, no shot should ever miss.
• Ignoring wrecking hits, the best quality hits (150%) should be approximately three times as good as the lowest quality hits (50%).
• Any wrecking hits I get should be approximately twice as good as the next best hit (300%).
• Because the both ships are stationary, the 250mm Railgun setups with and without TCs should come in almost identical - tracking speed should be irrelevant.
• Because both ships are stationary, the tests with MWD on should be almost identical to the ones with MWD off - signature resolution should be irrelevant.

Since every shot is hitting, we should also see that the average damage per hit is roughly in line with the base damage of each weapon (ammo damage x damage modifier, plus any skills and ship bonuses) minus anything that gets resisted. My kinetic and thermal resists on the Cormorant average out to about 37%, and the base damage of the guns with my skills comes out to 138 for the 425mms and 90 for the 250mms. So we should expect to see around 87 damage per shot for the 425mms, and 57 damage per shot for the 250mms. Note that the Naga's guns naturally put out more damage, so we're not looking to compare damage numbers between the two - what we're looking for is how the actual numbers compare to these predictions.

I ran each test for the first 100 shots. Here are the results:

Three of my tests included a wrecking hit. In these cases, I've included the best non-wrecking hit in brackets. Let's test my assumptions:

• No misses - Check
• Best non-wrecking hits roughly 3x worst hits - Check
• Wrecking hits roughly twice as good as best hits - Check
• 250mm with TCs no different to without - Check (in one case it did slightly better and in one case it did slightly worse, but the average hit was within a few points in both cases).
• MWD on no different to MWD off - Check (the 425mms actually did slightly better with it off, but the fact that the 250mms didn't tells me this is more likely down to luck and the relatively small sample size than anything else).

So far, we're 5 for 5. What about the average damage?

The Naga is obviously putting out more damage per hit than the Ferox, but if we compare this to the base damage for each gun we'll find that they both come in incredibly close to where we'd expect. The 425mms with MWD came out a little low, however every other result came through with the average damage only a few points away from my prediction of 87 for 425mms and 57 for 250mms.

Ideally I'd run this a few more times to get a better sample, but the results so far strongly indicate that both guns are performing optimally - they're both dealing their full base damage on average, with all non-wrecking hits distributed evenly between 50% and 150% which is exactly what the formula suggests. At this point I think it's safe to say that the mechanics are all performing as expected - tracking speed and signature resolution appear to make no significant difference, and all damage readings are roughly as predicted.

Of course this is just our baseline, however we've already disproved one of the premises which Ripard seems to hold:

Premise 3 - Signature resolution should have a significant impact if both ships are stationary.
 

Analysis: While one set of Naga results did come in slightly under the expected damage output, this was actually the one where the target's signature radius was larger than the gun's signature resolution (which is the opposite to what I assume Ripard would expect). When firing at the target with its MWD off, the Naga's rails actually performed almost exactly to their base damage output. This strongly suggests that high signature resolution relative to the target has no detrimental effect if both ships are stationary.

Conclusion: FALSE

To address the other premises, we need to try this against a moving target.


Doing it Live

For my main test, I set the Cormorant orbiting at 60km (well within optimal for both guns, removing range from the equation) with its MWD on. This gives it a speed of around 1.6km/s, and the same signature as before - 551m.

If Ripard is correct, I should see the Naga's 425mms close to or ahead of the Ferox's 250mms because they have a higher tracking speed. If signature resolution truly has no impact when people are MWDing around, the Ferox with TCs should perform almost identically to the Naga.

If I'm right, the Ferox with 250mms should handily beat the Naga - it's tracking is worse, but it's signature resolution is over three times smaller. The Ferox with TCs should completely wipe the floor with both.

Assuming the formula is correct and working is intended (which is partly what Ripard disputes), I should also see the following:

• Worst hit (ignoring misses) should not be not worse than the worst hit from my baseline (you can't hit for less than 50% of your base damage.
• The more a weapon system misses, the lower its best hit should be compared to my baseline (ignoring wrecking hits). This is because the best hits are the ones which become misses first.

Have you guessed what the result was yet?

Not only did the results come in in the order that I predicted, but the Ferox's 250mm rails completely trounced the Naga's 425mms in both cases. Of 100 shots from the Naga, only 16 hit at all - every other shot missed completely. The TC Ferox on the other hand (which has identical tracking to the Naga and only differs in signature resolution) hit 88 of its shots, doing nearly 6 times as much total damage as the Naga using a weapon system with about 30% less base damage than its larger counterpart.

The average hit for the Naga was still comparable, and that really comes back to the tracking formula

Let's check my two assumptions too:

• Worst hits not worse than 50% - Check
• Best hits proportional to number of misses - Check (the dual TC Ferox had best hits almost as good as its baseline, whereas the Naga's best hit was barely higher than its worst. The regular Ferox comes in between the two).

I have to be honest - I was certain that the Ferox would win even with worse tracking, but I'm actually really surprised by just how much better it was. Colour me impressed.

So where do we end up on Ripard's remaining four premises:


Premise 1 - A Naga with 425mms rails and faction ammo should out-track a Ferox with 250mms rails and spike by a significant margin.
 

Analysis: This is quite clearly not true, in fact the results show exactly the opposite - the standard Ferox came out leages ahead of the Naga, despite having a lower tracking speed.

Conclusion: FALSE

Premise 2 - Signature resolution should not have a significant impact on a ship which is MWDing around.
 

Analysis: The dual TC Ferox has a tracking speed almost identical to the Naga. Since both ships were inside optimal and all other factors were equal other than their base damage, this allows us to test the effect of signature resolution in isolation. The difference between these two results suggests that not only is signature resolution significant, but it's incredibly significant!
 

Conclusion: FALSE

 

Premise 4 - The math of the tracking formula is not functioning as intended.
 

Analysis: Over the course of this test, I've stated a number of assumptions which should have proved true providing the formula is functioning correctly. While I can't prove the validity of the formula outright, the results appear fully consistent with the stated formula and I see no reason to suspect that it is functioning in any way other than the one intended.
 

Conclusion: FALSE

 

Premise 5 - Ripard understands how signature resolution works.

Analysis: Clearly, you don't.

Conclusion: FALSE


A Final Word

To be fair to Ripard, I'd like to attribute to him one final premise. While he doesn't outright state this, I believe this is the core principle which has lead him to make the faulty conclusions above:

Premise 6 - Despite the effects of tracking and signature resolution, it is still possible for large guns to deal significant damage to small/fast targets under the right circumstances.

Analysis: While the overall damage output and hit rate for the Naga was abysmally low, the fact that no hit can score less than 50% of its base damage output (as well as the possibility of wrecking hits regardless of hit chance), it seems evident that the occasional hits which large guns do score can still be significant. The reason that they don't seem significant in practice is because you're not usually firing a single gun - you're firing up to 8 separate guns which all make their own roll, and you're only seeing the total figure. When 7 guns miss and one hits for 50%, the overall number makes it appear that turrets can hit for significantly less than this lower bound, which is not actually the case. It is also evident from the results that even a full target can be hit consistently if its angular velocity is low enough, meaning that multiple spread out shooters or poor piloting can lead to such small ships being hit far more often than you might otherwise expect.

Conclusion: TRUE


To Ripard himself: I apologise for calling you out on this, but I trust that as someone who regularly calls out people in positions of authority on just about anything, you won't hold it against me. I expect you'll be called out on many things over the course of the next year, so think of it as practice. If you'd like me to explain in any more detail I'd be happy to do so, but I suspect you won't! In before you tell me "that's what I was saying all along, you just misunderstood" - you sneaky politician you!

And to the readers: Despite how this post might appear, I actually have a lot of respect for Ripard and I think he has a very solid grasp of the game (even if he doesn't understand how signature resolution works). I'm confident that he'll make a good choice for CSM8, and definitely deserves your consideration when you cast your votes.


Update: I've had a few responses from Ripard on twitter. Unfortunately I think he's still missed the point, either that or he just doesn't want to admit he's wrong, being a politician and all!

Update 2: I've just had a second exchange with Ripard, and we've come to an understanding. Kudos to him for admitting the mistake - I'd say it bodes quite well for his CSM campaign!