Dev Log

[larsiusprime]

It's not too bad, but I've definitely got a scratchy throat. Here's hoping I can head this one off at the pass.

Progress will be a little slow today as I deal with logistics, email, PM's, forum stuff, PAX follow-ups, customer support, etc, but hopefully I'll be back in full swing and bugfixin' tomorrow morning.

I need to put together a new patch/update release schedule. My main concern is having enough time and testing between a new patch release candidate to discover if any *additional* bugs have been introduced into the latest release from fixing old ones. We used to just upload builds on friday, test over the weekend, and release monday, which was better than nothing. We might do that again, but it depends on how active our testers are. From the looks of it we've got plenty of people submitting bug reports, and if you all could stay active and help me with patch candidates, that'd be great.

Here's what I'm thinking for the new process:
-Look over bugs -> attempt to fix them -> generate latest "development" patch.
-Upload latest dev build to test server.
-Wait X days to see if there are new bugs introduced in dev build
-Fix stuff as needed, when build seems stable enough, upload it to the update server.

So, under this model we could release a new build each week and have another one waiting in the wings to be tested. So, here's what this week might look like:

Do an initial round of bug-fixing for the most important things immediately. Put that up ASAP on the test server. Call that Version 1.0.2. You guys hammer it with testing, and I stop making changes before Friday, any further fixes go into Version 1.0.3, which I keep working on. If version 1.0.2 doesn't introduce any NEW bugs, we release it on Monday and I keep working on Version 1.0.3 until I freeze work on that the next Friday, and start on 1.0.4, etc, etc.

Sound good? Any suggestions? I'm kind of new to the whole formal software development / release schedule stuff. I want something simple I can stick with but that still gives me enough time to catch horrible bugs I just introduced before they go live.

[coyot]

I can only mention that active regression testing would be a good idea. When you fix something, tell us whereever else is the code used or where it reaches. Otherwise you have no guarantee that in that particular week, enough people will exercise the touched part of the code to find any new bugs.

Edit: plus looking for edge cases to test. I.e. summon all defenders, use maximum characters for names, that sort of stuff.

[larsiusprime]

@coyot:

Sounds like a good idea. Still feeling pretty sick today, hard to concentrate on much besides writing emails and forum posts. Should be able to get back into coding soon, there's some bugs I'd really like to fix.

[larsiusprime]

Hey everybody!

Jumped back into bugfixing today. I moved a lot of the old bugs into the "fixed bug" forum just to clean things out so I could focus. Next, I've transcribed a couple of the bugs from the bug forum to bugzilla - if you're a tester, by all means please file things in Bugzilla as well! The bug forum is there basically just to provide another means for discussion, and to catch information that someone might not otherwise post if they had to go through the trouble of Bugzilla. This means we'll get duplicate information, sure, but I'd rather have too much info rather than too little.

So:
Best: file the bug in bugzilla
Good: post the bug on bugzilla AND the bug forum
Okay: post the bug just on the forum
Bad: don't post the bug at all

Anyways, I've started fixing some things here and there. I've updated these bugs in bugzilla and I'll get around to updating them on the bug forum eventually as well.

1) Fixed the star counting. There were two errors here in my star counting function not properly recognizing sidequests as battles for this purpose.
2) Fixed the knight-still-cleaves-when-paused bug: This was just bad animation data for the knight in data_defender.xml. This also fixes the "cleave is way too slow" bug.
3) Threw in a memory leak safeguard for repeatedly restarting battles without ever hitting the overworld (where memory is normally refreshed and cleared).

These fixes will be in version 1.0.2

I'm still investigating the big scary "game freezes when you summon everybody" bug. I'll see if I can repro that as it's one of my top priorities.

We will still be using the old test server, and I think now that we know the auto-updater works, I'm not going to mess with having test builds auto-update from a separate update server, that's just too much hassle and it's likely to introduce errors into shipping code. Instead, test builds will use the same update server as the live ones, and since their version number will be higher, they should not detect any auto-updates. This means you'll usually be installing these guys manually from the test server, so sorry about that. It's just way simpler for me that way and lets me spend more time fixing bugs and developing.

I will try to put up a windows build of version 1.0.2 by the end of the day. We are planning on deploying the kongregate version early next week, and I'll try to get the compatibility mode builds together late next week. If version 1.0.2 survives testing, we'll plan on rolling it out as a patch on Monday.

I'll let you know when the new build is up.

Thanks!

[larsiusprime]

New build is up on the test server. Also threw in a quick fix to make sure the csv data tables are recognized in mod mode. Once I can organize the bug database a bit better and stop being sick, I'll have a clearer head and start going by severity/etc, but for now I'm just kind of attacking things at random.

[Yinan]

didn't want to post a new Thread for that...

Wouldn't it be better if the game checked FIRST if there is a new update and loading assets, background and cutscenes AFTER that? Y bother loading the game when you download the new patch and apply it...

[larsiusprime]

@Yinan:

It needs to load fonts and the graphics required for the interface in order to display the update system. Of course, backgrounds & cutscenes could presumably be loaded afterwards to shorten the startup time. All the assets used to be instantaneously baked in before the size got too big to embed directly, and so the load sequence is carried over from back then.

[larsiusprime]

Okay, big update here guys:

I think that the "new" memory leak / crashing is related to the damage-queue bug. So, the whole thing about delayed hits was originally implemented to solve a visual / performance issue - ten bajillion hits all being resolved on the same frame. As it turns out, this isn't a big performance deal, and keeping a super long damage queue just makes it worse. Furthermore, it was leading to really weird issues with speed-independent damage resolution, which even led to porcupine damage resolving ultra-fast, and sometimes several seconds (or even minutes!) after the damage was originally done, resulting in super-dead berserkers.

So, here's the new solution. If too much damage happens at once, it still delays the hits, but only for one update cycle. Then, it takes all those delayed hits and unleashes them in the next update cycle, but it "congeals" their numbers together, so 10 simultaneous hits will just create one bouncing number with the sum of the damage. Congealed hits segregate by visual text type - so red will group with red, white with white, blue with blue, gold with gold, etc. Visually I'm not sure how much of a difference this will make since there's still 60 update cycles per second, but it should theoretically help performance. Another thing I can do is increase the "visual damage congeal" time (which is completely separate from hit resolution, it's just a display issue) so that all damage numbers over a second are "congealed" rather than all damage over one frame, or something in between. This way there might be some point in turning on damage numbers late game - right now it's just a sea of numbers that obscures everything you're doing so I always personally turn it off.

Anyway, the visual stuff is semi irrelevant. The most important thing is this solves several problems with the damage queue.

First, damage resolves in game time now, I plugged the glitch that made it resolve in real-time while doing the above fix. This should keep battles mostly deterministic on different speeds.

Second, it should give you credit for hits faster, so if you've put enough damage into the super-duper sheep to kill it, you don't have to wait 5 minutes or pause the game to run the 'damage clock' forward.

Third, berserkers seem to not get ultra-murdered by porcupine like they used to. This is because their damage now resolves in proportion to their animations and cooldowns, not via the damage queue. Although the damage queue would delay hits, sometimes even up to a minute, they would resolve one right after another, at a rate of one per update cycle, so when the berserkers hits finally landed, they'd all land way faster than the berserker could normally attack, so four or five full cooldown cycles of attacks would be dealt out over a single second, and he'd get hit by all the porcupine damage at once, and his regen and nearby healers would have no chance to save him.

I found some other bugs while I was at it, too. For one, porcupine damage was sometimes hitting the wrong target, because the enemy wasn't always keeping perfect track of who had hit them with the melee attack. (Ie, a berserker hits a thorned enemy, but before the porcupine resolves a ranger's arrow hits them, so the porcupine attacks the ranger!) This is now fixed.

Next, I found some "invisible poison" being applied randomly when testing the super sheep level. Turned out that Ketta had 0 points in "poison shot" which was not being ignored - instead it was just giving her a poison damage flavor with 0 amount of poison. This would resolve so that it would show the poison effect on the enemy sprite, but not in the preview panel, and would do 0 damage every second. Now I just make it explicitly ignore passive skills with 0 points in them.

So, the thing to look out for with fixing the damage queue is that I've changed the way damage resolves. If I made a mistake, this means that hits could be getting dropped, ignored, or otherwise folded, spindled, or mutilated. I don't *think* I did any of that, but if you guys could keep a close eye on things for me, that'd be much appreciated.

To aid in debugging stuff like this in the future, in the next few patches I will try to eventually create a "battle log" that you can have the game write out to, so we can examine it to make sure nothing bad is happening. If anyone has any suggestions for how to format that or what data it should capture, I'm all ears.

Finally, the next test version will be 1.0.21. If I had been smart, the first patch would have been 1.0.01 so I could have counted up from there, but I forgot the leading zero. I also forgot to correct my mistake with the second test version and just do 1.0.11. At any rate, I'm fixing this now to give myself more room. I'm not entirely sure how the internal Adobe AIR version checker works, but I know that 1.0.21 is unambiguously "bigger" than 1.0.2, regardless of how it's reckoning things, so that's where we're going from here.

I'll let you know when the new version is up.

[larsiusprime]

Okay guys, here's what I've gotten fixed so far in 1.0.21:

-Damage queue bug
-Invisible poison
-Incorrect porcupine targets
-Psi Shield doesn't end battle
-Spawn wave block 1-frame glitch
-Removed delay before first enemy spawn in wave

So, using Psi Shield will no longer keep the battle from ending, and I fixed an annoying visual glitch where the incoming wave block would pop all the way over the left for a single frame before the spawning starts. Also, now the first enemy in a wave will spawn as soon as the block lights up, not after an initial waiting period. Before, the waiting period between individual spawns was also happening before the first enemy hit the map.

Compiled version 1.0.21 right now, it's going up to the test server as we speak (windows build only for now). It should be ready in about 10 minutes.

My primary goal is to stabilize this build, make sure it doesn't ADD any new bugs, and see whether fixing the damage queue also fixes the crashes. If those are all true, then 1.0.21 will be what ships on Monday.

[coyot]

To aid in debugging stuff like this in the future, in the next few patches I will try to eventually create a "battle log" that you can have the game write out to, so we can examine it to make sure nothing bad is happening. If anyone has any suggestions for how to format that or what data it should capture, I'm all ears.

Hmm, it's probably gonna end up chewing tons of disk space . But the full debug battle log should capture:
1) enemy spawns (start points AND spawns-on-death)
2) player actions (place defender, boost defender, cast spell, optionally also disable/enable skill although that's not likely to be needed)
3) all attack actions, ideally with targetting logic included, minimally intended target for projectile attacks (as there might be a big delay in the log between activation and hit, and this would be super useful for issues like attacks at something that gets killed in the meantime)
4) processing of all attacks in both directions (defender to creep and vice versa)

I imagine that all defenders would be labelled (keyboard invocation code and current boost level would be good and short), all creeps would be labelled by wave/type/instance. Every attack should list:

- always target starting HP
- always attack flavor
- always base skill damage
- if present, multiplier from defender (zeal),
- if present, multiplier from equipment (unique item bonus)
- if triggered, multiplier from critical/rage
- if armor is present, armor pierce effect
- if armor is still present, armor reduction of damage (is this the right place to take armor into account?)
- if present, multiplier from debuff (bleed)
- if present, multiplier from dmg resist
- actual damage applied, remaning health of the enemy
- any hitback action, depending on how it's processed. Is it just another attack that gets "queued"? Or is it processed immediately?
- for every applicable trait, calculated chance to activate (just to verify that the number is correct and the trait was processed)
- for every trait that passed the roll, effect application (is immunity to a particular effect calculated before this step or the previous one? Immunity getting applied should definitely get logged)
- for attacks by creeps/bosses, save action from uniques if applicable

Rinse and repeat for all targets hit by the attack/splash.

I can imagine that this would generate really huge amounts of data, even if the logging would be as condensed as possible (basically for every item, you'd need at least separator, short code (1-2 chars) and value. If you could stick the attack to 80 chars or something, readable one line, it'd be nice.

Minimal example:
QA_1_Rev03/15_HPS1000_D60_HPF940 (Slak, attack 1 on revenant wave 3 number 15, starting hp 1000, base damage 60, final HP 940 - no traits or resistances or whatever)
TA_2_Kni04/01_HPS2000_D200_ - and here begins the fun, because the knight needs to calculate armor pierce, damage to substract, armor break to apply, check for knockback vs immunity.... and that's excluding buffs and debuffs...

But I can imagine this getting quite complicated i.e. with lasting area attacks and creeps walking into them.

So, this sounds very much like a project for a long winter night or three and I don't think you'll want to go there unless utterly bored OR totally fooled by some nasty bug in damage handling...

[Nepene]

Thank you for all the bug fixes, it's good to see annoying issues quickly be resolves and in such a clear and coherent manner.

[coyot]

Lars, dragon fire still fails to set enemies on Fire reliably. Just observed it in Niru+Wrenna sidequest on advanced. First battle since game launch, first few waves, no fire resistant enemy - and yet I got almost nobody. A few waves later, about 50:50.
I have a feeling that it might have to do with some timing issues.

Also, in that same map, I've observed Wrenna's attacks sometimes fail to deliver 3x damage to splitters and worms (in fact fairly often) wearing the right weapon. I've got one working hypothesis - if the attack hits multiple targets, then on the first non-worm it turns off the bonus. I've never observed Ice Ball fail, Sleet failed almost always as did Blizzard.
But there might be a separate bug with Ice Splash - that I've NEVER seen do 3x dmg to worms.

Best reproduced in Night of the Living Knights on extreme, if you mod this map - replace the first and last abomination with a HUGE level Fallen knight (so that he will be invincible due to high armor) - and observe how sleet and blizzard fail to do crit dmg to the remaining abominations.

[larsiusprime]

Looking into it, thanks for the update.

[larsiusprime]

This took several hours of continuously digging on the same repro case to figure out, because once I had an ideal repro setup, it would still only happen about 80% of the time. As it turns out, it's related to the remnant of the old damage queue. If an enemy has been hit already (by DOT damage, presumably) on the same frame they take dragon fire, the dragon fire gets delayed by a frame, and the flavors aren't stored correctly, so the damage gets through next frame, but not the burn.

Now that I've found the point of error I should be able to fix it.

UPDATE:
Found it. The reason the flavors being dropped was because I forgot to store a deep copy of the flavor list in the "DamageBucket" struct for the delayed hit. Instead, I was just storing a local reference, and when that list of flavors on the original attack gets destructed, the reference in the DamageBucket gets destructed too. So, next frame when the delayed hit lands, it deals damage, but no burn.

Doing a final stress test to make sure. I'll run the repro setup after the fix, roll it back, and the apply it agin.

Results:
After the fix: 3 runthroughs, no bug.
Roll it back: 3 runthroughs, bug happens 3 times.
Re-apply the fix: 3 runthroughs, no bug.

So I'm gonna call that fixed. Fix will be in version 1.0.23, which I'm uploading to the test servers later today.

I believe some of the other problems you might have seen might be related to this "flavors sometimes dropped" bug - extra damage against certain enemy types is implemented by a selectively-targeted critical hit with 100% chance of success. So, if that flavor gets dropped much like dragon fire's burn, sometimes it will deal 3x damage and sometimes it won't.

The AOE seems to be a separate case of the weapon's flavor not propagating into the AOE attack's flavors.

[coyot]

If you can push a build within an hour, I'll repeat my plays from yesterday where it showed more or less reliably.

[larsiusprime]

The problems with Wrenna's staff seem to be fixed now, too. The problem was related to flavors, once again. Now abominations always take 3x critical damage, whether it's ice ball, sleet, blizzard, ice splash, ice shard, anything. At least on my tests, that is. Gonna compile this up and put it on the test server as 1.0.23 - should be up in 15 minutes.

[coyot]

Great.. the list's getting shorter!

Which of the pending bugs you consider showstoppers?

* the permastunned enemies/other problems with pushback (including a bit weird graphical glitch - if an enemy is hit in mid-air, the "jumping" damage number follows him in a weird fashion... hard to repro and minor, just thought it worth mentioning).
* crash/lock on restarting battle
* disappearing APCs (well, it makes the game easier, so not a showstopper, and it's a pain to repro - but I've had it happen about 3-4 times already)
* regression of the invincible mad cultist being too fast. Have you solved it in the past just by triggering the dialogue sooner?

[larsiusprime]

The crashes are my biggest priority. I'd like to fix permastunned enemies, because I'm going to wager that can keep a battle from finishing.

Is the invincible cultist still killing people in the latest build? The last bug I saw related to that was a typo in one of Zelemir's skills. If that's still happening I'll put it as high priority.

Disappearing APC's is something I'd like to fix if we can get it to happen consistently, but it's overall lower on my priority list.

[coyot]

Only partial success:
Dragon fire seems to work fine.
Wrenna's Ice splash projectile itself now deals critical against worms from weapon bonus properly, but the loss of crit flavour for multi-hit attacks remains, be it Ice shard or AoEs.

Best reproduced on Halls of God King Advanced. First two waves from red spawn are normal revenant and worm. If you kill the revenant by magic and other units, recall them and place Wrenna there just for the worm wave, you'll only ever see critical numbers from all her attacks including splash from Ice Splash.

If you restart and summon just wrenna, a bit away from the exit so that the worms catch up with the revenant, you'll suddenly see some criticals on worms, but quite often non-criticals, especially with Sleet and Blizzard - but it can be also reproduced with Ice Shard. If you could make a few custom 1creep waves with small delays/careful spacing, i.e. 1 rev 1 worm 1 rev 1 worm 1 rev, and hit them from front with just ice shard, you'll see that no crit will happen. If you zap the first rev with magic, you'll get crit on first worm and normal on second. (I haven't tried that but I'll bet a beer on it)

[larsiusprime]

Well, glad we made some progress at least I'll check on the splash problems - I could have sworn I saw them working, but maybe I missed something. Thanks again for these great repro cases, it makes it a lot easier to fix the bugs.